2023-02-11 20:00:13,884 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524 2023-02-11 20:00:13,899 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-02-11 20:00:13,913 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-11 20:00:13,914 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748, deleteOnExit=true 2023-02-11 20:00:13,914 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-11 20:00:13,914 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/test.cache.data in system properties and HBase conf 2023-02-11 20:00:13,915 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/hadoop.tmp.dir in system properties and HBase conf 2023-02-11 20:00:13,915 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/hadoop.log.dir in system properties and HBase conf 2023-02-11 20:00:13,916 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-11 20:00:13,916 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-11 20:00:13,916 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-11 20:00:14,040 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-11 20:00:14,426 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-11 20:00:14,431 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-11 20:00:14,432 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-11 20:00:14,432 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-11 20:00:14,432 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 20:00:14,432 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-11 20:00:14,433 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-11 20:00:14,433 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 20:00:14,433 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 20:00:14,434 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-11 20:00:14,434 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/nfs.dump.dir in system properties and HBase conf 2023-02-11 20:00:14,435 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/java.io.tmpdir in system properties and HBase conf 2023-02-11 20:00:14,435 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 20:00:14,435 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-11 20:00:14,436 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-11 20:00:14,959 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 20:00:14,963 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 20:00:15,244 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-11 20:00:15,411 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-11 20:00:15,425 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 20:00:15,461 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 20:00:15,496 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/java.io.tmpdir/Jetty_localhost_44411_hdfs____.rw2cg2/webapp 2023-02-11 20:00:15,676 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44411 2023-02-11 20:00:15,730 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 20:00:15,731 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 20:00:16,205 WARN [Listener at localhost/43583] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 20:00:16,344 WARN [Listener at localhost/43583] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 20:00:16,370 WARN [Listener at localhost/43583] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 20:00:16,378 INFO [Listener at localhost/43583] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 20:00:16,385 INFO [Listener at localhost/43583] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/java.io.tmpdir/Jetty_localhost_32837_datanode____.rs5x30/webapp 2023-02-11 20:00:16,518 INFO [Listener at localhost/43583] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:32837 2023-02-11 20:00:16,910 WARN [Listener at localhost/38765] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 20:00:16,951 WARN [Listener at localhost/38765] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 20:00:16,956 WARN [Listener at localhost/38765] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 20:00:16,958 INFO [Listener at localhost/38765] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 20:00:16,967 INFO [Listener at localhost/38765] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/java.io.tmpdir/Jetty_localhost_43423_datanode____.ers2o3/webapp 2023-02-11 20:00:17,085 INFO [Listener at localhost/38765] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43423 2023-02-11 20:00:17,108 WARN [Listener at localhost/34059] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 20:00:17,141 WARN [Listener at localhost/34059] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 20:00:17,148 WARN [Listener at localhost/34059] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 20:00:17,150 INFO [Listener at localhost/34059] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 20:00:17,157 INFO [Listener at localhost/34059] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/java.io.tmpdir/Jetty_localhost_41873_datanode____.azq78i/webapp 2023-02-11 20:00:17,283 INFO [Listener at localhost/34059] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41873 2023-02-11 20:00:17,297 WARN [Listener at localhost/33691] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 20:00:17,450 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa4de03bca501e30: Processing first storage report for DS-abd549bb-85dd-44fc-89c2-6e4c02dd0c22 from datanode c1a54257-2d0a-45a8-ba80-a5d503ce840b 2023-02-11 20:00:17,451 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa4de03bca501e30: from storage DS-abd549bb-85dd-44fc-89c2-6e4c02dd0c22 node DatanodeRegistration(127.0.0.1:43501, datanodeUuid=c1a54257-2d0a-45a8-ba80-a5d503ce840b, infoPort=40995, infoSecurePort=0, ipcPort=34059, storageInfo=lv=-57;cid=testClusterID;nsid=486880303;c=1676145615035), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-11 20:00:17,452 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x7dbecd6ff6eb7cea: Processing first storage report for DS-7d53001e-1634-4d6d-be5d-d9cc59035ca2 from datanode 6c4dbc0d-7a4b-4326-b825-e2c9ec77a985 2023-02-11 20:00:17,452 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x7dbecd6ff6eb7cea: from storage DS-7d53001e-1634-4d6d-be5d-d9cc59035ca2 node DatanodeRegistration(127.0.0.1:39807, datanodeUuid=6c4dbc0d-7a4b-4326-b825-e2c9ec77a985, infoPort=32991, infoSecurePort=0, ipcPort=38765, storageInfo=lv=-57;cid=testClusterID;nsid=486880303;c=1676145615035), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 20:00:17,452 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3935fee008663e27: Processing first storage report for DS-8dce3fed-4370-49e1-b69a-9fb7c1c1dc64 from datanode 6172956c-8b21-46b3-8a3e-55f2d221c88d 2023-02-11 20:00:17,452 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3935fee008663e27: from storage DS-8dce3fed-4370-49e1-b69a-9fb7c1c1dc64 node DatanodeRegistration(127.0.0.1:34289, datanodeUuid=6172956c-8b21-46b3-8a3e-55f2d221c88d, infoPort=34715, infoSecurePort=0, ipcPort=33691, storageInfo=lv=-57;cid=testClusterID;nsid=486880303;c=1676145615035), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 20:00:17,452 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa4de03bca501e30: Processing first storage report for DS-afde3750-3adf-4b9e-ad6f-cbdf801f9e3b from datanode c1a54257-2d0a-45a8-ba80-a5d503ce840b 2023-02-11 20:00:17,453 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa4de03bca501e30: from storage DS-afde3750-3adf-4b9e-ad6f-cbdf801f9e3b node DatanodeRegistration(127.0.0.1:43501, datanodeUuid=c1a54257-2d0a-45a8-ba80-a5d503ce840b, infoPort=40995, infoSecurePort=0, ipcPort=34059, storageInfo=lv=-57;cid=testClusterID;nsid=486880303;c=1676145615035), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-11 20:00:17,453 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x7dbecd6ff6eb7cea: Processing first storage report for DS-d767beab-d9bb-49ce-8121-81061ba0c631 from datanode 6c4dbc0d-7a4b-4326-b825-e2c9ec77a985 2023-02-11 20:00:17,453 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x7dbecd6ff6eb7cea: from storage DS-d767beab-d9bb-49ce-8121-81061ba0c631 node DatanodeRegistration(127.0.0.1:39807, datanodeUuid=6c4dbc0d-7a4b-4326-b825-e2c9ec77a985, infoPort=32991, infoSecurePort=0, ipcPort=38765, storageInfo=lv=-57;cid=testClusterID;nsid=486880303;c=1676145615035), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 20:00:17,453 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3935fee008663e27: Processing first storage report for DS-711b4d19-b079-4e1b-8cf4-c57fccde5b12 from datanode 6172956c-8b21-46b3-8a3e-55f2d221c88d 2023-02-11 20:00:17,453 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3935fee008663e27: from storage DS-711b4d19-b079-4e1b-8cf4-c57fccde5b12 node DatanodeRegistration(127.0.0.1:34289, datanodeUuid=6172956c-8b21-46b3-8a3e-55f2d221c88d, infoPort=34715, infoSecurePort=0, ipcPort=33691, storageInfo=lv=-57;cid=testClusterID;nsid=486880303;c=1676145615035), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 20:00:17,767 DEBUG [Listener at localhost/33691] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524 2023-02-11 20:00:17,827 INFO [Listener at localhost/33691] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748/zookeeper_0, clientPort=51732, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-11 20:00:17,840 INFO [Listener at localhost/33691] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=51732 2023-02-11 20:00:17,851 INFO [Listener at localhost/33691] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 20:00:17,854 INFO [Listener at localhost/33691] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 20:00:18,570 INFO [Listener at localhost/33691] util.FSUtils(479): Created version file at hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5 with version=8 2023-02-11 20:00:18,571 INFO [Listener at localhost/33691] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/hbase-staging 2023-02-11 20:00:18,869 INFO [Listener at localhost/33691] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-11 20:00:19,313 INFO [Listener at localhost/33691] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 20:00:19,344 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 20:00:19,345 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 20:00:19,346 INFO [Listener at localhost/33691] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 20:00:19,346 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 20:00:19,346 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 20:00:19,495 INFO [Listener at localhost/33691] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 20:00:19,565 DEBUG [Listener at localhost/33691] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-11 20:00:19,654 INFO [Listener at localhost/33691] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39971 2023-02-11 20:00:19,664 INFO [Listener at localhost/33691] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 20:00:19,666 INFO [Listener at localhost/33691] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 20:00:19,686 INFO [Listener at localhost/33691] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39971 connecting to ZooKeeper ensemble=127.0.0.1:51732 2023-02-11 20:00:19,724 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:399710x0, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 20:00:19,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:39971-0x10044f8555a0000 connected 2023-02-11 20:00:19,750 DEBUG [Listener at localhost/33691] zookeeper.ZKUtil(164): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 20:00:19,751 DEBUG [Listener at localhost/33691] zookeeper.ZKUtil(164): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 20:00:19,754 DEBUG [Listener at localhost/33691] zookeeper.ZKUtil(164): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 20:00:19,762 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39971 2023-02-11 20:00:19,762 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39971 2023-02-11 20:00:19,763 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39971 2023-02-11 20:00:19,764 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39971 2023-02-11 20:00:19,764 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39971 2023-02-11 20:00:19,771 INFO [Listener at localhost/33691] master.HMaster(439): hbase.rootdir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5, hbase.cluster.distributed=false 2023-02-11 20:00:19,842 INFO [Listener at localhost/33691] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 20:00:19,842 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 20:00:19,843 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 20:00:19,843 INFO [Listener at localhost/33691] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 20:00:19,843 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 20:00:19,843 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 20:00:19,848 INFO [Listener at localhost/33691] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 20:00:19,851 INFO [Listener at localhost/33691] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42011 2023-02-11 20:00:19,853 INFO [Listener at localhost/33691] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 20:00:19,860 INFO [Listener at localhost/33691] mob.MobFileCache(124): MobFileCache disabled 2023-02-11 20:00:19,861 INFO [Listener at localhost/33691] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 20:00:19,863 INFO [Listener at localhost/33691] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 20:00:19,865 INFO [Listener at localhost/33691] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42011 connecting to ZooKeeper ensemble=127.0.0.1:51732 2023-02-11 20:00:19,868 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:420110x0, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 20:00:19,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:42011-0x10044f8555a0001 connected 2023-02-11 20:00:19,869 DEBUG [Listener at localhost/33691] zookeeper.ZKUtil(164): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 20:00:19,871 DEBUG [Listener at localhost/33691] zookeeper.ZKUtil(164): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 20:00:19,872 DEBUG [Listener at localhost/33691] zookeeper.ZKUtil(164): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 20:00:19,874 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42011 2023-02-11 20:00:19,874 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42011 2023-02-11 20:00:19,875 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42011 2023-02-11 20:00:19,875 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42011 2023-02-11 20:00:19,876 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42011 2023-02-11 20:00:19,888 INFO [Listener at localhost/33691] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 20:00:19,888 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 20:00:19,888 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 20:00:19,888 INFO [Listener at localhost/33691] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 20:00:19,889 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 20:00:19,889 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 20:00:19,889 INFO [Listener at localhost/33691] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 20:00:19,890 INFO [Listener at localhost/33691] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35937 2023-02-11 20:00:19,891 INFO [Listener at localhost/33691] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 20:00:19,891 INFO [Listener at localhost/33691] mob.MobFileCache(124): MobFileCache disabled 2023-02-11 20:00:19,892 INFO [Listener at localhost/33691] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 20:00:19,895 INFO [Listener at localhost/33691] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 20:00:19,898 INFO [Listener at localhost/33691] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35937 connecting to ZooKeeper ensemble=127.0.0.1:51732 2023-02-11 20:00:19,902 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:359370x0, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 20:00:19,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:35937-0x10044f8555a0002 connected 2023-02-11 20:00:19,903 DEBUG [Listener at localhost/33691] zookeeper.ZKUtil(164): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 20:00:19,904 DEBUG [Listener at localhost/33691] zookeeper.ZKUtil(164): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 20:00:19,905 DEBUG [Listener at localhost/33691] zookeeper.ZKUtil(164): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 20:00:19,906 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35937 2023-02-11 20:00:19,908 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35937 2023-02-11 20:00:19,912 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35937 2023-02-11 20:00:19,912 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35937 2023-02-11 20:00:19,912 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35937 2023-02-11 20:00:19,924 INFO [Listener at localhost/33691] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 20:00:19,924 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 20:00:19,924 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 20:00:19,925 INFO [Listener at localhost/33691] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 20:00:19,925 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 20:00:19,925 INFO [Listener at localhost/33691] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 20:00:19,925 INFO [Listener at localhost/33691] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 20:00:19,929 INFO [Listener at localhost/33691] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44175 2023-02-11 20:00:19,929 INFO [Listener at localhost/33691] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 20:00:19,930 INFO [Listener at localhost/33691] mob.MobFileCache(124): MobFileCache disabled 2023-02-11 20:00:19,931 INFO [Listener at localhost/33691] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 20:00:19,932 INFO [Listener at localhost/33691] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 20:00:19,933 INFO [Listener at localhost/33691] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:44175 connecting to ZooKeeper ensemble=127.0.0.1:51732 2023-02-11 20:00:19,936 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:441750x0, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 20:00:19,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:44175-0x10044f8555a0003 connected 2023-02-11 20:00:19,938 DEBUG [Listener at localhost/33691] zookeeper.ZKUtil(164): regionserver:44175-0x10044f8555a0003, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 20:00:19,938 DEBUG [Listener at localhost/33691] zookeeper.ZKUtil(164): regionserver:44175-0x10044f8555a0003, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 20:00:19,939 DEBUG [Listener at localhost/33691] zookeeper.ZKUtil(164): regionserver:44175-0x10044f8555a0003, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 20:00:19,941 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44175 2023-02-11 20:00:19,941 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44175 2023-02-11 20:00:19,941 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44175 2023-02-11 20:00:19,944 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44175 2023-02-11 20:00:19,944 DEBUG [Listener at localhost/33691] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44175 2023-02-11 20:00:19,948 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,39971,1676145618710 2023-02-11 20:00:19,957 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 20:00:19,959 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,39971,1676145618710 2023-02-11 20:00:19,979 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 20:00:19,979 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:44175-0x10044f8555a0003, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 20:00:19,979 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 20:00:19,979 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 20:00:19,979 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 20:00:19,981 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 20:00:19,982 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,39971,1676145618710 from backup master directory 2023-02-11 20:00:19,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 20:00:19,986 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,39971,1676145618710 2023-02-11 20:00:19,986 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 20:00:19,987 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 20:00:19,987 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,39971,1676145618710 2023-02-11 20:00:19,990 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-11 20:00:19,991 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-11 20:00:20,080 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/hbase.id with ID: 0ea4476c-2113-4db5-990f-ad82c252efdb 2023-02-11 20:00:20,119 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 20:00:20,132 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 20:00:20,179 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x1813e35a to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:20,212 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56953f78, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:20,236 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-11 20:00:20,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-11 20:00:20,255 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-11 20:00:20,255 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-11 20:00:20,257 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-11 20:00:20,262 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-11 20:00:20,263 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 20:00:20,295 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/MasterData/data/master/store-tmp 2023-02-11 20:00:20,331 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:20,332 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 20:00:20,332 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 20:00:20,332 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 20:00:20,332 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 20:00:20,332 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 20:00:20,332 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 20:00:20,332 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676145620331Disabling compacts and flushes for region at 1676145620332 (+1 ms)Disabling writes for close at 1676145620332Writing region close event to WAL at 1676145620332Closed at 1676145620332 2023-02-11 20:00:20,333 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/MasterData/WALs/jenkins-hbase4.apache.org,39971,1676145618710 2023-02-11 20:00:20,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39971%2C1676145618710, suffix=, logDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/MasterData/WALs/jenkins-hbase4.apache.org,39971,1676145618710, archiveDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/MasterData/oldWALs, maxLogs=10 2023-02-11 20:00:20,412 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39807,DS-7d53001e-1634-4d6d-be5d-d9cc59035ca2,DISK] 2023-02-11 20:00:20,412 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43501,DS-abd549bb-85dd-44fc-89c2-6e4c02dd0c22,DISK] 2023-02-11 20:00:20,412 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34289,DS-8dce3fed-4370-49e1-b69a-9fb7c1c1dc64,DISK] 2023-02-11 20:00:20,421 DEBUG [RS-EventLoopGroup-5-1] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-11 20:00:20,495 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/MasterData/WALs/jenkins-hbase4.apache.org,39971,1676145618710/jenkins-hbase4.apache.org%2C39971%2C1676145618710.1676145620362 2023-02-11 20:00:20,495 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43501,DS-abd549bb-85dd-44fc-89c2-6e4c02dd0c22,DISK], DatanodeInfoWithStorage[127.0.0.1:34289,DS-8dce3fed-4370-49e1-b69a-9fb7c1c1dc64,DISK], DatanodeInfoWithStorage[127.0.0.1:39807,DS-7d53001e-1634-4d6d-be5d-d9cc59035ca2,DISK]] 2023-02-11 20:00:20,496 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-11 20:00:20,496 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:20,499 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 20:00:20,500 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 20:00:20,567 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-11 20:00:20,574 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-11 20:00:20,601 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-11 20:00:20,613 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:20,618 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 20:00:20,620 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 20:00:20,632 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 20:00:20,636 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 20:00:20,637 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63822873, jitterRate=-0.048965081572532654}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:20,637 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676145620526Initializing all the Stores at 1676145620529 (+3 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676145620530 (+1 ms)Cleaning up temporary data from old regions at 1676145620625 (+95 ms)Cleaning up detritus from prior splits at 1676145620626 (+1 ms)Region opened successfully at 1676145620637 (+11 ms) 2023-02-11 20:00:20,638 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-11 20:00:20,658 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-11 20:00:20,658 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-11 20:00:20,662 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-11 20:00:20,664 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-11 20:00:20,701 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 36 msec 2023-02-11 20:00:20,701 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-11 20:00:20,733 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-11 20:00:20,739 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-11 20:00:20,763 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-11 20:00:20,767 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-11 20:00:20,768 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-11 20:00:20,773 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-11 20:00:20,777 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-11 20:00:20,780 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 20:00:20,781 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-11 20:00:20,782 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-11 20:00:20,800 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-11 20:00:20,806 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 20:00:20,807 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 20:00:20,807 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 20:00:20,807 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 20:00:20,806 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:44175-0x10044f8555a0003, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 20:00:20,811 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,39971,1676145618710, sessionid=0x10044f8555a0000, setting cluster-up flag (Was=false) 2023-02-11 20:00:20,828 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 20:00:20,838 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-11 20:00:20,840 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39971,1676145618710 2023-02-11 20:00:20,846 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 20:00:20,852 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-11 20:00:20,854 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39971,1676145618710 2023-02-11 20:00:20,856 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp 2023-02-11 20:00:20,948 INFO [RS:2;jenkins-hbase4:44175] regionserver.HRegionServer(952): ClusterId : 0ea4476c-2113-4db5-990f-ad82c252efdb 2023-02-11 20:00:20,949 INFO [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer(952): ClusterId : 0ea4476c-2113-4db5-990f-ad82c252efdb 2023-02-11 20:00:20,949 INFO [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer(952): ClusterId : 0ea4476c-2113-4db5-990f-ad82c252efdb 2023-02-11 20:00:20,955 DEBUG [RS:2;jenkins-hbase4:44175] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 20:00:20,955 DEBUG [RS:0;jenkins-hbase4:42011] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 20:00:20,955 DEBUG [RS:1;jenkins-hbase4:35937] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 20:00:20,964 DEBUG [RS:1;jenkins-hbase4:35937] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 20:00:20,964 DEBUG [RS:2;jenkins-hbase4:44175] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 20:00:20,964 DEBUG [RS:0;jenkins-hbase4:42011] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 20:00:20,965 DEBUG [RS:2;jenkins-hbase4:44175] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 20:00:20,964 DEBUG [RS:1;jenkins-hbase4:35937] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 20:00:20,965 DEBUG [RS:0;jenkins-hbase4:42011] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 20:00:20,971 DEBUG [RS:2;jenkins-hbase4:44175] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 20:00:20,972 DEBUG [RS:1;jenkins-hbase4:35937] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 20:00:20,972 DEBUG [RS:2;jenkins-hbase4:44175] zookeeper.ReadOnlyZKClient(139): Connect 0x77c8214c to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:20,973 DEBUG [RS:0;jenkins-hbase4:42011] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 20:00:20,975 DEBUG [RS:0;jenkins-hbase4:42011] zookeeper.ReadOnlyZKClient(139): Connect 0x4ac087cc to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:20,975 DEBUG [RS:1;jenkins-hbase4:35937] zookeeper.ReadOnlyZKClient(139): Connect 0x4df21505 to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:21,017 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-11 20:00:21,017 DEBUG [RS:0;jenkins-hbase4:42011] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@440760e4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:21,018 DEBUG [RS:0;jenkins-hbase4:42011] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4038ecbc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 20:00:21,020 DEBUG [RS:1;jenkins-hbase4:35937] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4335f2cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:21,021 DEBUG [RS:1;jenkins-hbase4:35937] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@470a7e75, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 20:00:21,021 DEBUG [RS:2;jenkins-hbase4:44175] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@790fc1c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:21,022 DEBUG [RS:2;jenkins-hbase4:44175] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d304a3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 20:00:21,030 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 20:00:21,030 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 20:00:21,030 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 20:00:21,030 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 20:00:21,030 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-11 20:00:21,030 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,030 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 20:00:21,030 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,036 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676145651036 2023-02-11 20:00:21,039 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-11 20:00:21,042 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 20:00:21,043 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-11 20:00:21,044 DEBUG [RS:2;jenkins-hbase4:44175] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:44175 2023-02-11 20:00:21,046 DEBUG [RS:0;jenkins-hbase4:42011] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:42011 2023-02-11 20:00:21,046 DEBUG [RS:1;jenkins-hbase4:35937] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:35937 2023-02-11 20:00:21,049 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 20:00:21,050 INFO [RS:2;jenkins-hbase4:44175] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 20:00:21,050 INFO [RS:0;jenkins-hbase4:42011] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 20:00:21,051 INFO [RS:0;jenkins-hbase4:42011] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 20:00:21,050 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-11 20:00:21,050 INFO [RS:2;jenkins-hbase4:44175] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 20:00:21,051 DEBUG [RS:2;jenkins-hbase4:44175] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 20:00:21,050 INFO [RS:1;jenkins-hbase4:35937] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 20:00:21,052 INFO [RS:1;jenkins-hbase4:35937] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 20:00:21,052 DEBUG [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 20:00:21,051 DEBUG [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 20:00:21,056 INFO [RS:2;jenkins-hbase4:44175] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39971,1676145618710 with isa=jenkins-hbase4.apache.org/172.31.14.131:44175, startcode=1676145619924 2023-02-11 20:00:21,056 INFO [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39971,1676145618710 with isa=jenkins-hbase4.apache.org/172.31.14.131:42011, startcode=1676145619841 2023-02-11 20:00:21,056 INFO [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39971,1676145618710 with isa=jenkins-hbase4.apache.org/172.31.14.131:35937, startcode=1676145619887 2023-02-11 20:00:21,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-11 20:00:21,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-11 20:00:21,061 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-11 20:00:21,061 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-11 20:00:21,064 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,069 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-11 20:00:21,071 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-11 20:00:21,071 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-11 20:00:21,075 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-11 20:00:21,076 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-11 20:00:21,079 DEBUG [RS:1;jenkins-hbase4:35937] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 20:00:21,079 DEBUG [RS:2;jenkins-hbase4:44175] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 20:00:21,080 DEBUG [RS:0;jenkins-hbase4:42011] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 20:00:21,084 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676145621083,5,FailOnTimeoutGroup] 2023-02-11 20:00:21,088 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676145621085,5,FailOnTimeoutGroup] 2023-02-11 20:00:21,088 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,089 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-11 20:00:21,090 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,090 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,113 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 20:00:21,114 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 20:00:21,115 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5 2023-02-11 20:00:21,161 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44759, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 20:00:21,161 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59451, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 20:00:21,161 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50439, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 20:00:21,189 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:21,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:21,189 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39971] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:21,190 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39971] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:21,192 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 20:00:21,194 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/info 2023-02-11 20:00:21,195 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 20:00:21,196 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:21,196 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 20:00:21,199 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/rep_barrier 2023-02-11 20:00:21,199 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 20:00:21,200 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:21,202 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 20:00:21,204 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/table 2023-02-11 20:00:21,205 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 20:00:21,206 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:21,207 DEBUG [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5 2023-02-11 20:00:21,207 DEBUG [RS:2;jenkins-hbase4:44175] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5 2023-02-11 20:00:21,207 DEBUG [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5 2023-02-11 20:00:21,207 DEBUG [RS:2;jenkins-hbase4:44175] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:43583 2023-02-11 20:00:21,207 DEBUG [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:43583 2023-02-11 20:00:21,208 DEBUG [RS:2;jenkins-hbase4:44175] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 20:00:21,208 DEBUG [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:43583 2023-02-11 20:00:21,208 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740 2023-02-11 20:00:21,208 DEBUG [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 20:00:21,208 DEBUG [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 20:00:21,209 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740 2023-02-11 20:00:21,213 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 20:00:21,215 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 20:00:21,218 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 20:00:21,219 DEBUG [RS:2;jenkins-hbase4:44175] zookeeper.ZKUtil(162): regionserver:44175-0x10044f8555a0003, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:21,219 WARN [RS:2;jenkins-hbase4:44175] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 20:00:21,219 DEBUG [RS:0;jenkins-hbase4:42011] zookeeper.ZKUtil(162): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:21,219 INFO [RS:2;jenkins-hbase4:44175] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 20:00:21,219 DEBUG [RS:2;jenkins-hbase4:44175] regionserver.HRegionServer(1947): logDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/WALs/jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:21,219 WARN [RS:0;jenkins-hbase4:42011] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 20:00:21,225 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 20:00:21,225 DEBUG [RS:1;jenkins-hbase4:35937] zookeeper.ZKUtil(162): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:21,225 INFO [RS:0;jenkins-hbase4:42011] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 20:00:21,225 WARN [RS:1;jenkins-hbase4:35937] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 20:00:21,226 INFO [RS:1;jenkins-hbase4:35937] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 20:00:21,226 DEBUG [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer(1947): logDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/WALs/jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:21,226 DEBUG [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer(1947): logDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/WALs/jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:21,227 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72369474, jitterRate=0.07838919758796692}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 20:00:21,227 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676145621189Initializing all the Stores at 1676145621191 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676145621191Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676145621191Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676145621191Cleaning up temporary data from old regions at 1676145621210 (+19 ms)Cleaning up detritus from prior splits at 1676145621211 (+1 ms)Region opened successfully at 1676145621227 (+16 ms) 2023-02-11 20:00:21,227 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,44175,1676145619924] 2023-02-11 20:00:21,227 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 20:00:21,227 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35937,1676145619887] 2023-02-11 20:00:21,228 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 20:00:21,228 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42011,1676145619841] 2023-02-11 20:00:21,228 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 20:00:21,229 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 20:00:21,229 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 20:00:21,237 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 20:00:21,237 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676145621227Disabling compacts and flushes for region at 1676145621227Disabling writes for close at 1676145621229 (+2 ms)Writing region close event to WAL at 1676145621237 (+8 ms)Closed at 1676145621237 2023-02-11 20:00:21,246 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 20:00:21,246 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-11 20:00:21,247 DEBUG [RS:1;jenkins-hbase4:35937] zookeeper.ZKUtil(162): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:21,247 DEBUG [RS:0;jenkins-hbase4:42011] zookeeper.ZKUtil(162): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:21,247 DEBUG [RS:2;jenkins-hbase4:44175] zookeeper.ZKUtil(162): regionserver:44175-0x10044f8555a0003, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:21,248 DEBUG [RS:1;jenkins-hbase4:35937] zookeeper.ZKUtil(162): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:21,248 DEBUG [RS:2;jenkins-hbase4:44175] zookeeper.ZKUtil(162): regionserver:44175-0x10044f8555a0003, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:21,248 DEBUG [RS:0;jenkins-hbase4:42011] zookeeper.ZKUtil(162): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:21,248 DEBUG [RS:1;jenkins-hbase4:35937] zookeeper.ZKUtil(162): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:21,249 DEBUG [RS:0;jenkins-hbase4:42011] zookeeper.ZKUtil(162): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:21,249 DEBUG [RS:2;jenkins-hbase4:44175] zookeeper.ZKUtil(162): regionserver:44175-0x10044f8555a0003, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:21,255 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-11 20:00:21,260 DEBUG [RS:0;jenkins-hbase4:42011] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 20:00:21,260 DEBUG [RS:1;jenkins-hbase4:35937] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 20:00:21,260 DEBUG [RS:2;jenkins-hbase4:44175] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 20:00:21,269 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-11 20:00:21,270 INFO [RS:2;jenkins-hbase4:44175] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 20:00:21,270 INFO [RS:0;jenkins-hbase4:42011] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 20:00:21,270 INFO [RS:1;jenkins-hbase4:35937] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 20:00:21,273 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:21,293 INFO [RS:1;jenkins-hbase4:35937] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 20:00:21,293 INFO [RS:0;jenkins-hbase4:42011] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 20:00:21,297 INFO [RS:2;jenkins-hbase4:44175] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 20:00:21,299 INFO [RS:2;jenkins-hbase4:44175] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 20:00:21,299 INFO [RS:1;jenkins-hbase4:35937] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 20:00:21,299 INFO [RS:2;jenkins-hbase4:44175] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,299 INFO [RS:0;jenkins-hbase4:42011] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 20:00:21,299 INFO [RS:1;jenkins-hbase4:35937] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,300 INFO [RS:0;jenkins-hbase4:42011] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,301 INFO [RS:2;jenkins-hbase4:44175] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 20:00:21,301 INFO [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 20:00:21,301 INFO [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 20:00:21,310 INFO [RS:1;jenkins-hbase4:35937] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,310 INFO [RS:2;jenkins-hbase4:44175] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,310 DEBUG [RS:1;jenkins-hbase4:35937] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,310 DEBUG [RS:2;jenkins-hbase4:44175] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,310 DEBUG [RS:1;jenkins-hbase4:35937] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,310 DEBUG [RS:2;jenkins-hbase4:44175] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,310 INFO [RS:0;jenkins-hbase4:42011] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,311 DEBUG [RS:2;jenkins-hbase4:44175] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,310 DEBUG [RS:1;jenkins-hbase4:35937] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,311 DEBUG [RS:2;jenkins-hbase4:44175] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,311 DEBUG [RS:0;jenkins-hbase4:42011] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,311 DEBUG [RS:2;jenkins-hbase4:44175] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,311 DEBUG [RS:1;jenkins-hbase4:35937] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,311 DEBUG [RS:2;jenkins-hbase4:44175] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 20:00:21,311 DEBUG [RS:1;jenkins-hbase4:35937] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,311 DEBUG [RS:2;jenkins-hbase4:44175] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,311 DEBUG [RS:1;jenkins-hbase4:35937] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 20:00:21,311 DEBUG [RS:0;jenkins-hbase4:42011] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,311 DEBUG [RS:1;jenkins-hbase4:35937] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,311 DEBUG [RS:0;jenkins-hbase4:42011] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,311 DEBUG [RS:1;jenkins-hbase4:35937] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,311 DEBUG [RS:0;jenkins-hbase4:42011] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,311 DEBUG [RS:2;jenkins-hbase4:44175] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,312 DEBUG [RS:0;jenkins-hbase4:42011] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,312 DEBUG [RS:2;jenkins-hbase4:44175] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,312 DEBUG [RS:0;jenkins-hbase4:42011] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 20:00:21,312 DEBUG [RS:2;jenkins-hbase4:44175] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,312 DEBUG [RS:1;jenkins-hbase4:35937] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,312 DEBUG [RS:0;jenkins-hbase4:42011] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,312 DEBUG [RS:1;jenkins-hbase4:35937] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,312 DEBUG [RS:0;jenkins-hbase4:42011] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,313 DEBUG [RS:0;jenkins-hbase4:42011] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,313 DEBUG [RS:0;jenkins-hbase4:42011] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 20:00:21,314 INFO [RS:1;jenkins-hbase4:35937] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,314 INFO [RS:2;jenkins-hbase4:44175] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,314 INFO [RS:1;jenkins-hbase4:35937] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,314 INFO [RS:2;jenkins-hbase4:44175] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,314 INFO [RS:1;jenkins-hbase4:35937] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,314 INFO [RS:2;jenkins-hbase4:44175] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,314 INFO [RS:0;jenkins-hbase4:42011] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,314 INFO [RS:0;jenkins-hbase4:42011] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,314 INFO [RS:0;jenkins-hbase4:42011] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,333 INFO [RS:1;jenkins-hbase4:35937] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 20:00:21,336 INFO [RS:1;jenkins-hbase4:35937] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35937,1676145619887-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,336 INFO [RS:0;jenkins-hbase4:42011] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 20:00:21,337 INFO [RS:2;jenkins-hbase4:44175] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 20:00:21,337 INFO [RS:0;jenkins-hbase4:42011] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42011,1676145619841-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,337 INFO [RS:2;jenkins-hbase4:44175] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44175,1676145619924-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,360 INFO [RS:0;jenkins-hbase4:42011] regionserver.Replication(203): jenkins-hbase4.apache.org,42011,1676145619841 started 2023-02-11 20:00:21,360 INFO [RS:1;jenkins-hbase4:35937] regionserver.Replication(203): jenkins-hbase4.apache.org,35937,1676145619887 started 2023-02-11 20:00:21,360 INFO [RS:2;jenkins-hbase4:44175] regionserver.Replication(203): jenkins-hbase4.apache.org,44175,1676145619924 started 2023-02-11 20:00:21,360 INFO [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,42011,1676145619841, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42011, sessionid=0x10044f8555a0001 2023-02-11 20:00:21,360 INFO [RS:2;jenkins-hbase4:44175] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,44175,1676145619924, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:44175, sessionid=0x10044f8555a0003 2023-02-11 20:00:21,360 INFO [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,35937,1676145619887, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35937, sessionid=0x10044f8555a0002 2023-02-11 20:00:21,361 DEBUG [RS:2;jenkins-hbase4:44175] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 20:00:21,361 DEBUG [RS:0;jenkins-hbase4:42011] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 20:00:21,361 DEBUG [RS:1;jenkins-hbase4:35937] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 20:00:21,361 DEBUG [RS:1;jenkins-hbase4:35937] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:21,361 DEBUG [RS:2;jenkins-hbase4:44175] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:21,361 DEBUG [RS:1;jenkins-hbase4:35937] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35937,1676145619887' 2023-02-11 20:00:21,361 DEBUG [RS:0;jenkins-hbase4:42011] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:21,362 DEBUG [RS:0;jenkins-hbase4:42011] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42011,1676145619841' 2023-02-11 20:00:21,362 DEBUG [RS:0;jenkins-hbase4:42011] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 20:00:21,362 DEBUG [RS:1;jenkins-hbase4:35937] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 20:00:21,361 DEBUG [RS:2;jenkins-hbase4:44175] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44175,1676145619924' 2023-02-11 20:00:21,362 DEBUG [RS:2;jenkins-hbase4:44175] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 20:00:21,363 DEBUG [RS:1;jenkins-hbase4:35937] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 20:00:21,363 DEBUG [RS:2;jenkins-hbase4:44175] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 20:00:21,363 DEBUG [RS:0;jenkins-hbase4:42011] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 20:00:21,363 DEBUG [RS:1;jenkins-hbase4:35937] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 20:00:21,363 DEBUG [RS:1;jenkins-hbase4:35937] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 20:00:21,364 DEBUG [RS:1;jenkins-hbase4:35937] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:21,364 DEBUG [RS:1;jenkins-hbase4:35937] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35937,1676145619887' 2023-02-11 20:00:21,364 DEBUG [RS:1;jenkins-hbase4:35937] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 20:00:21,364 DEBUG [RS:0;jenkins-hbase4:42011] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 20:00:21,364 DEBUG [RS:0;jenkins-hbase4:42011] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 20:00:21,364 DEBUG [RS:2;jenkins-hbase4:44175] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 20:00:21,364 DEBUG [RS:0;jenkins-hbase4:42011] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:21,365 DEBUG [RS:1;jenkins-hbase4:35937] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 20:00:21,364 DEBUG [RS:2;jenkins-hbase4:44175] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 20:00:21,365 DEBUG [RS:0;jenkins-hbase4:42011] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42011,1676145619841' 2023-02-11 20:00:21,365 DEBUG [RS:0;jenkins-hbase4:42011] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 20:00:21,365 DEBUG [RS:2;jenkins-hbase4:44175] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:21,365 DEBUG [RS:2;jenkins-hbase4:44175] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44175,1676145619924' 2023-02-11 20:00:21,365 DEBUG [RS:2;jenkins-hbase4:44175] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 20:00:21,365 DEBUG [RS:1;jenkins-hbase4:35937] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 20:00:21,365 INFO [RS:1;jenkins-hbase4:35937] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 20:00:21,366 INFO [RS:1;jenkins-hbase4:35937] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 20:00:21,366 DEBUG [RS:2;jenkins-hbase4:44175] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 20:00:21,366 DEBUG [RS:2;jenkins-hbase4:44175] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 20:00:21,366 INFO [RS:2;jenkins-hbase4:44175] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 20:00:21,367 DEBUG [RS:0;jenkins-hbase4:42011] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 20:00:21,367 INFO [RS:2;jenkins-hbase4:44175] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 20:00:21,368 DEBUG [RS:0;jenkins-hbase4:42011] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 20:00:21,368 INFO [RS:0;jenkins-hbase4:42011] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 20:00:21,368 INFO [RS:0;jenkins-hbase4:42011] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 20:00:21,425 DEBUG [jenkins-hbase4:39971] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-11 20:00:21,428 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 20:00:21,435 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 20:00:21,435 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 20:00:21,435 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 20:00:21,435 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 20:00:21,438 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35937,1676145619887, state=OPENING 2023-02-11 20:00:21,445 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-11 20:00:21,446 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 20:00:21,447 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 20:00:21,450 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:21,479 INFO [RS:0;jenkins-hbase4:42011] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42011%2C1676145619841, suffix=, logDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/WALs/jenkins-hbase4.apache.org,42011,1676145619841, archiveDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/oldWALs, maxLogs=32 2023-02-11 20:00:21,480 INFO [RS:1;jenkins-hbase4:35937] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35937%2C1676145619887, suffix=, logDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/WALs/jenkins-hbase4.apache.org,35937,1676145619887, archiveDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/oldWALs, maxLogs=32 2023-02-11 20:00:21,479 INFO [RS:2;jenkins-hbase4:44175] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44175%2C1676145619924, suffix=, logDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/WALs/jenkins-hbase4.apache.org,44175,1676145619924, archiveDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/oldWALs, maxLogs=32 2023-02-11 20:00:21,524 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43501,DS-abd549bb-85dd-44fc-89c2-6e4c02dd0c22,DISK] 2023-02-11 20:00:21,525 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34289,DS-8dce3fed-4370-49e1-b69a-9fb7c1c1dc64,DISK] 2023-02-11 20:00:21,527 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39807,DS-7d53001e-1634-4d6d-be5d-d9cc59035ca2,DISK] 2023-02-11 20:00:21,527 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43501,DS-abd549bb-85dd-44fc-89c2-6e4c02dd0c22,DISK] 2023-02-11 20:00:21,532 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39807,DS-7d53001e-1634-4d6d-be5d-d9cc59035ca2,DISK] 2023-02-11 20:00:21,532 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34289,DS-8dce3fed-4370-49e1-b69a-9fb7c1c1dc64,DISK] 2023-02-11 20:00:21,533 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43501,DS-abd549bb-85dd-44fc-89c2-6e4c02dd0c22,DISK] 2023-02-11 20:00:21,533 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34289,DS-8dce3fed-4370-49e1-b69a-9fb7c1c1dc64,DISK] 2023-02-11 20:00:21,533 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39807,DS-7d53001e-1634-4d6d-be5d-d9cc59035ca2,DISK] 2023-02-11 20:00:21,558 INFO [RS:0;jenkins-hbase4:42011] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/WALs/jenkins-hbase4.apache.org,42011,1676145619841/jenkins-hbase4.apache.org%2C42011%2C1676145619841.1676145621486 2023-02-11 20:00:21,558 INFO [RS:2;jenkins-hbase4:44175] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/WALs/jenkins-hbase4.apache.org,44175,1676145619924/jenkins-hbase4.apache.org%2C44175%2C1676145619924.1676145621487 2023-02-11 20:00:21,558 INFO [RS:1;jenkins-hbase4:35937] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/WALs/jenkins-hbase4.apache.org,35937,1676145619887/jenkins-hbase4.apache.org%2C35937%2C1676145619887.1676145621486 2023-02-11 20:00:21,558 DEBUG [RS:0;jenkins-hbase4:42011] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34289,DS-8dce3fed-4370-49e1-b69a-9fb7c1c1dc64,DISK], DatanodeInfoWithStorage[127.0.0.1:39807,DS-7d53001e-1634-4d6d-be5d-d9cc59035ca2,DISK], DatanodeInfoWithStorage[127.0.0.1:43501,DS-abd549bb-85dd-44fc-89c2-6e4c02dd0c22,DISK]] 2023-02-11 20:00:21,559 DEBUG [RS:1;jenkins-hbase4:35937] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43501,DS-abd549bb-85dd-44fc-89c2-6e4c02dd0c22,DISK], DatanodeInfoWithStorage[127.0.0.1:34289,DS-8dce3fed-4370-49e1-b69a-9fb7c1c1dc64,DISK], DatanodeInfoWithStorage[127.0.0.1:39807,DS-7d53001e-1634-4d6d-be5d-d9cc59035ca2,DISK]] 2023-02-11 20:00:21,559 DEBUG [RS:2;jenkins-hbase4:44175] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34289,DS-8dce3fed-4370-49e1-b69a-9fb7c1c1dc64,DISK], DatanodeInfoWithStorage[127.0.0.1:39807,DS-7d53001e-1634-4d6d-be5d-d9cc59035ca2,DISK], DatanodeInfoWithStorage[127.0.0.1:43501,DS-abd549bb-85dd-44fc-89c2-6e4c02dd0c22,DISK]] 2023-02-11 20:00:21,645 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:21,649 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 20:00:21,653 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36188, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 20:00:21,666 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-11 20:00:21,666 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 20:00:21,671 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35937%2C1676145619887.meta, suffix=.meta, logDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/WALs/jenkins-hbase4.apache.org,35937,1676145619887, archiveDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/oldWALs, maxLogs=32 2023-02-11 20:00:21,700 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43501,DS-abd549bb-85dd-44fc-89c2-6e4c02dd0c22,DISK] 2023-02-11 20:00:21,702 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34289,DS-8dce3fed-4370-49e1-b69a-9fb7c1c1dc64,DISK] 2023-02-11 20:00:21,702 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39807,DS-7d53001e-1634-4d6d-be5d-d9cc59035ca2,DISK] 2023-02-11 20:00:21,711 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/WALs/jenkins-hbase4.apache.org,35937,1676145619887/jenkins-hbase4.apache.org%2C35937%2C1676145619887.meta.1676145621673.meta 2023-02-11 20:00:21,714 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43501,DS-abd549bb-85dd-44fc-89c2-6e4c02dd0c22,DISK], DatanodeInfoWithStorage[127.0.0.1:34289,DS-8dce3fed-4370-49e1-b69a-9fb7c1c1dc64,DISK], DatanodeInfoWithStorage[127.0.0.1:39807,DS-7d53001e-1634-4d6d-be5d-d9cc59035ca2,DISK]] 2023-02-11 20:00:21,715 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-11 20:00:21,716 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-11 20:00:21,733 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-11 20:00:21,739 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-11 20:00:21,746 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-11 20:00:21,746 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:21,746 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-11 20:00:21,747 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-11 20:00:21,750 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 20:00:21,752 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/info 2023-02-11 20:00:21,752 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/info 2023-02-11 20:00:21,753 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 20:00:21,753 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:21,754 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 20:00:21,755 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/rep_barrier 2023-02-11 20:00:21,755 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/rep_barrier 2023-02-11 20:00:21,756 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 20:00:21,757 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:21,757 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 20:00:21,759 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/table 2023-02-11 20:00:21,759 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/table 2023-02-11 20:00:21,760 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 20:00:21,761 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:21,763 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740 2023-02-11 20:00:21,766 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740 2023-02-11 20:00:21,769 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 20:00:21,772 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 20:00:21,773 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62341959, jitterRate=-0.07103241980075836}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 20:00:21,774 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676145621747Writing region info on filesystem at 1676145621747Initializing all the Stores at 1676145621749 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676145621750 (+1 ms)Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676145621750Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676145621750Cleaning up temporary data from old regions at 1676145621768 (+18 ms)Cleaning up detritus from prior splits at 1676145621768Running coprocessor post-open hooks at 1676145621773 (+5 ms)Region opened successfully at 1676145621773 2023-02-11 20:00:21,785 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676145621634 2023-02-11 20:00:21,806 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-11 20:00:21,807 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-11 20:00:21,807 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35937,1676145619887, state=OPEN 2023-02-11 20:00:21,811 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-11 20:00:21,811 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 20:00:21,816 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-11 20:00:21,816 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35937,1676145619887 in 361 msec 2023-02-11 20:00:21,821 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-11 20:00:21,821 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 562 msec 2023-02-11 20:00:21,828 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 924 msec 2023-02-11 20:00:21,828 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676145619975, completionTime=-1 2023-02-11 20:00:21,829 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-11 20:00:21,829 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-11 20:00:21,898 DEBUG [hconnection-0x6065ab53-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:21,903 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36190, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:21,926 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-11 20:00:21,926 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676145681926 2023-02-11 20:00:21,926 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676145741926 2023-02-11 20:00:21,926 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 97 msec 2023-02-11 20:00:21,953 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39971,1676145618710-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,953 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39971,1676145618710-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,953 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39971,1676145618710-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,956 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:39971, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,956 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-11 20:00:21,967 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-11 20:00:21,980 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-11 20:00:21,982 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 20:00:21,994 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-11 20:00:21,997 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 20:00:22,001 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 20:00:22,026 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/hbase/namespace/77102a65aff4927797da7a30e0b8de4a 2023-02-11 20:00:22,029 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/hbase/namespace/77102a65aff4927797da7a30e0b8de4a empty. 2023-02-11 20:00:22,030 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/hbase/namespace/77102a65aff4927797da7a30e0b8de4a 2023-02-11 20:00:22,030 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-11 20:00:22,070 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-11 20:00:22,073 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 77102a65aff4927797da7a30e0b8de4a, NAME => 'hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:22,096 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:22,096 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 77102a65aff4927797da7a30e0b8de4a, disabling compactions & flushes 2023-02-11 20:00:22,097 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a. 2023-02-11 20:00:22,097 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a. 2023-02-11 20:00:22,097 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a. after waiting 0 ms 2023-02-11 20:00:22,097 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a. 2023-02-11 20:00:22,097 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a. 2023-02-11 20:00:22,097 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 77102a65aff4927797da7a30e0b8de4a: Waiting for close lock at 1676145622096Disabling compacts and flushes for region at 1676145622096Disabling writes for close at 1676145622097 (+1 ms)Writing region close event to WAL at 1676145622097Closed at 1676145622097 2023-02-11 20:00:22,102 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 20:00:22,125 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676145622106"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145622106"}]},"ts":"1676145622106"} 2023-02-11 20:00:22,165 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-11 20:00:22,167 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 20:00:22,174 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145622168"}]},"ts":"1676145622168"} 2023-02-11 20:00:22,181 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-11 20:00:22,187 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 20:00:22,188 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 20:00:22,189 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 20:00:22,189 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 20:00:22,189 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 20:00:22,192 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=77102a65aff4927797da7a30e0b8de4a, ASSIGN}] 2023-02-11 20:00:22,197 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=77102a65aff4927797da7a30e0b8de4a, ASSIGN 2023-02-11 20:00:22,207 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=77102a65aff4927797da7a30e0b8de4a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42011,1676145619841; forceNewPlan=false, retain=false 2023-02-11 20:00:22,360 INFO [jenkins-hbase4:39971] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-11 20:00:22,361 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=77102a65aff4927797da7a30e0b8de4a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:22,361 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676145622361"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145622361"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145622361"}]},"ts":"1676145622361"} 2023-02-11 20:00:22,367 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 77102a65aff4927797da7a30e0b8de4a, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:22,523 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:22,523 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 20:00:22,526 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54114, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 20:00:22,532 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a. 2023-02-11 20:00:22,532 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 77102a65aff4927797da7a30e0b8de4a, NAME => 'hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a.', STARTKEY => '', ENDKEY => ''} 2023-02-11 20:00:22,533 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 77102a65aff4927797da7a30e0b8de4a 2023-02-11 20:00:22,533 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:22,533 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 77102a65aff4927797da7a30e0b8de4a 2023-02-11 20:00:22,533 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 77102a65aff4927797da7a30e0b8de4a 2023-02-11 20:00:22,535 INFO [StoreOpener-77102a65aff4927797da7a30e0b8de4a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 77102a65aff4927797da7a30e0b8de4a 2023-02-11 20:00:22,539 DEBUG [StoreOpener-77102a65aff4927797da7a30e0b8de4a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/namespace/77102a65aff4927797da7a30e0b8de4a/info 2023-02-11 20:00:22,539 DEBUG [StoreOpener-77102a65aff4927797da7a30e0b8de4a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/namespace/77102a65aff4927797da7a30e0b8de4a/info 2023-02-11 20:00:22,539 INFO [StoreOpener-77102a65aff4927797da7a30e0b8de4a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 77102a65aff4927797da7a30e0b8de4a columnFamilyName info 2023-02-11 20:00:22,540 INFO [StoreOpener-77102a65aff4927797da7a30e0b8de4a-1] regionserver.HStore(310): Store=77102a65aff4927797da7a30e0b8de4a/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:22,544 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/namespace/77102a65aff4927797da7a30e0b8de4a 2023-02-11 20:00:22,545 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/namespace/77102a65aff4927797da7a30e0b8de4a 2023-02-11 20:00:22,550 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 77102a65aff4927797da7a30e0b8de4a 2023-02-11 20:00:22,554 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/namespace/77102a65aff4927797da7a30e0b8de4a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 20:00:22,555 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 77102a65aff4927797da7a30e0b8de4a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72236681, jitterRate=0.07641042768955231}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:22,555 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 77102a65aff4927797da7a30e0b8de4a: Running coprocessor pre-open hook at 1676145622533Writing region info on filesystem at 1676145622533Initializing all the Stores at 1676145622535 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676145622535Cleaning up temporary data from old regions at 1676145622546 (+11 ms)Cleaning up detritus from prior splits at 1676145622546Running coprocessor post-open hooks at 1676145622555 (+9 ms)Region opened successfully at 1676145622555 2023-02-11 20:00:22,557 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a., pid=6, masterSystemTime=1676145622522 2023-02-11 20:00:22,562 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a. 2023-02-11 20:00:22,563 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a. 2023-02-11 20:00:22,564 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=77102a65aff4927797da7a30e0b8de4a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:22,565 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676145622564"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145622564"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145622564"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145622564"}]},"ts":"1676145622564"} 2023-02-11 20:00:22,572 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-11 20:00:22,572 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 77102a65aff4927797da7a30e0b8de4a, server=jenkins-hbase4.apache.org,42011,1676145619841 in 201 msec 2023-02-11 20:00:22,576 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-11 20:00:22,577 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=77102a65aff4927797da7a30e0b8de4a, ASSIGN in 380 msec 2023-02-11 20:00:22,579 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 20:00:22,579 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145622579"}]},"ts":"1676145622579"} 2023-02-11 20:00:22,582 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-11 20:00:22,586 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 20:00:22,589 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 603 msec 2023-02-11 20:00:22,599 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-11 20:00:22,601 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-11 20:00:22,601 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 20:00:22,623 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:22,627 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54120, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:22,646 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-11 20:00:22,667 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 20:00:22,674 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 39 msec 2023-02-11 20:00:22,681 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-11 20:00:22,699 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 20:00:22,705 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 23 msec 2023-02-11 20:00:22,717 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-11 20:00:22,720 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-11 20:00:22,720 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.733sec 2023-02-11 20:00:22,723 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-11 20:00:22,725 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-11 20:00:22,725 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-11 20:00:22,726 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39971,1676145618710-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-11 20:00:22,728 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39971,1676145618710-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-11 20:00:22,737 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-11 20:00:22,754 DEBUG [Listener at localhost/33691] zookeeper.ReadOnlyZKClient(139): Connect 0x5b210801 to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:22,760 DEBUG [Listener at localhost/33691] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@da5f7e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:22,778 DEBUG [hconnection-0x7415b278-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:22,791 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36196, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:22,802 INFO [Listener at localhost/33691] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,39971,1676145618710 2023-02-11 20:00:22,833 INFO [Listener at localhost/33691] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=362, OpenFileDescriptor=695, MaxFileDescriptor=60000, SystemLoadAverage=399, ProcessCount=168, AvailableMemoryMB=4412 2023-02-11 20:00:22,847 DEBUG [Listener at localhost/33691] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-11 20:00:22,851 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39862, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-11 20:00:22,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'testRestoreSnapshot-1676145622835', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-11 20:00:22,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:22,874 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676145622835 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 20:00:22,878 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676145622835 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 20:00:22,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1676145622835" procId is: 9 2023-02-11 20:00:22,888 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:22,888 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:22,889 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 empty. 2023-02-11 20:00:22,889 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 empty. 2023-02-11 20:00:22,892 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:22,892 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:22,893 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:22,900 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e empty. 2023-02-11 20:00:22,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-11 20:00:22,900 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:22,900 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:22,900 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:22,901 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:22,901 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 empty. 2023-02-11 20:00:22,903 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 empty. 2023-02-11 20:00:22,903 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 empty. 2023-02-11 20:00:22,904 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:22,904 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:22,904 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:22,904 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676145622835 regions 2023-02-11 20:00:22,935 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/.tabledesc/.tableinfo.0000000001 2023-02-11 20:00:22,937 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(7671): creating {ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676145622835', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:22,938 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(7671): creating {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676145622835', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:22,938 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(7671): creating {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676145622835', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:23,034 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:23,035 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(1603): Closing 7c040ebb400ff6a8415e55ad85bbbe0e, disabling compactions & flushes 2023-02-11 20:00:23,035 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:23,035 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:23,035 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. after waiting 0 ms 2023-02-11 20:00:23,035 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:23,035 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:23,035 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(1557): Region close journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Waiting for close lock at 1676145623035Disabling compacts and flushes for region at 1676145623035Disabling writes for close at 1676145623035Writing region close event to WAL at 1676145623035Closed at 1676145623035 2023-02-11 20:00:23,036 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(7671): creating {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676145622835', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:23,038 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:23,039 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(1603): Closing f1fc9bf98611ec0266597565431cc2a1, disabling compactions & flushes 2023-02-11 20:00:23,039 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:23,039 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:23,039 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. after waiting 0 ms 2023-02-11 20:00:23,039 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:23,039 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:23,039 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(1557): Region close journal for f1fc9bf98611ec0266597565431cc2a1: Waiting for close lock at 1676145623039Disabling compacts and flushes for region at 1676145623039Disabling writes for close at 1676145623039Writing region close event to WAL at 1676145623039Closed at 1676145623039 2023-02-11 20:00:23,040 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(7671): creating {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676145622835', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:23,042 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:23,043 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(1603): Closing db772e5c38666094ec2e3b8fb5e805c2, disabling compactions & flushes 2023-02-11 20:00:23,043 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:23,043 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:23,043 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. after waiting 0 ms 2023-02-11 20:00:23,043 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:23,043 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:23,043 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(1557): Region close journal for db772e5c38666094ec2e3b8fb5e805c2: Waiting for close lock at 1676145623043Disabling compacts and flushes for region at 1676145623043Disabling writes for close at 1676145623043Writing region close event to WAL at 1676145623043Closed at 1676145623043 2023-02-11 20:00:23,044 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(7671): creating {ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676145622835', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:23,101 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:23,102 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:23,102 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(1603): Closing c630b1709e678e0b00c070c7cac86d30, disabling compactions & flushes 2023-02-11 20:00:23,104 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(1603): Closing 1a4676c43c8662115dfd922e9ff72451, disabling compactions & flushes 2023-02-11 20:00:23,105 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:23,105 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:23,105 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:23,105 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:23,105 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:23,105 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. after waiting 0 ms 2023-02-11 20:00:23,105 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(1603): Closing 8de911ca888e28adfca434ff163cbba9, disabling compactions & flushes 2023-02-11 20:00:23,105 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. after waiting 0 ms 2023-02-11 20:00:23,105 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:23,105 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:23,105 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:23,105 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:23,105 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. after waiting 0 ms 2023-02-11 20:00:23,105 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:23,105 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:23,105 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-0] regionserver.HRegion(1557): Region close journal for 1a4676c43c8662115dfd922e9ff72451: Waiting for close lock at 1676145623104Disabling compacts and flushes for region at 1676145623104Disabling writes for close at 1676145623105 (+1 ms)Writing region close event to WAL at 1676145623105Closed at 1676145623105 2023-02-11 20:00:23,105 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:23,105 INFO [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:23,106 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-2] regionserver.HRegion(1557): Region close journal for c630b1709e678e0b00c070c7cac86d30: Waiting for close lock at 1676145623102Disabling compacts and flushes for region at 1676145623102Disabling writes for close at 1676145623105 (+3 ms)Writing region close event to WAL at 1676145623105Closed at 1676145623105 2023-02-11 20:00:23,106 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676145622835-pool-1] regionserver.HRegion(1557): Region close journal for 8de911ca888e28adfca434ff163cbba9: Waiting for close lock at 1676145623105Disabling compacts and flushes for region at 1676145623105Disabling writes for close at 1676145623105Writing region close event to WAL at 1676145623105Closed at 1676145623105 2023-02-11 20:00:23,110 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676145622835 execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 20:00:23,112 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145623111"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145623111"}]},"ts":"1676145623111"} 2023-02-11 20:00:23,112 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145623111"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145623111"}]},"ts":"1676145623111"} 2023-02-11 20:00:23,112 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145623111"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145623111"}]},"ts":"1676145623111"} 2023-02-11 20:00:23,112 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145623111"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145623111"}]},"ts":"1676145623111"} 2023-02-11 20:00:23,112 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145623111"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145623111"}]},"ts":"1676145623111"} 2023-02-11 20:00:23,112 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145623111"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145623111"}]},"ts":"1676145623111"} 2023-02-11 20:00:23,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-11 20:00:23,161 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 20:00:23,162 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676145622835 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 20:00:23,163 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145623163"}]},"ts":"1676145623163"} 2023-02-11 20:00:23,165 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=ENABLING in hbase:meta 2023-02-11 20:00:23,170 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 20:00:23,175 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 20:00:23,175 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 20:00:23,175 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 20:00:23,175 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 20:00:23,175 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN}] 2023-02-11 20:00:23,179 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN 2023-02-11 20:00:23,179 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN 2023-02-11 20:00:23,180 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN 2023-02-11 20:00:23,180 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN 2023-02-11 20:00:23,182 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42011,1676145619841; forceNewPlan=false, retain=false 2023-02-11 20:00:23,182 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN 2023-02-11 20:00:23,182 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44175,1676145619924; forceNewPlan=false, retain=false 2023-02-11 20:00:23,182 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35937,1676145619887; forceNewPlan=false, retain=false 2023-02-11 20:00:23,182 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44175,1676145619924; forceNewPlan=false, retain=false 2023-02-11 20:00:23,183 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN 2023-02-11 20:00:23,183 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35937,1676145619887; forceNewPlan=false, retain=false 2023-02-11 20:00:23,185 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42011,1676145619841; forceNewPlan=false, retain=false 2023-02-11 20:00:23,332 INFO [jenkins-hbase4:39971] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 20:00:23,335 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:23,335 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:23,335 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:23,335 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:23,335 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:23,335 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145623335"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145623335"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145623335"}]},"ts":"1676145623335"} 2023-02-11 20:00:23,335 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145623335"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145623335"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145623335"}]},"ts":"1676145623335"} 2023-02-11 20:00:23,335 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145623335"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145623335"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145623335"}]},"ts":"1676145623335"} 2023-02-11 20:00:23,335 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145623335"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145623335"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145623335"}]},"ts":"1676145623335"} 2023-02-11 20:00:23,335 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145623335"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145623335"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145623335"}]},"ts":"1676145623335"} 2023-02-11 20:00:23,339 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=14, state=RUNNABLE; OpenRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:23,341 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE; OpenRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:23,343 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=13, state=RUNNABLE; OpenRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:23,346 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=10, state=RUNNABLE; OpenRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:23,346 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:23,347 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145623346"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145623346"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145623346"}]},"ts":"1676145623346"} 2023-02-11 20:00:23,347 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=15, state=RUNNABLE; OpenRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:23,350 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=11, state=RUNNABLE; OpenRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:23,495 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:23,495 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 20:00:23,497 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36922, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 20:00:23,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:23,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 20:00:23,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:23,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:23,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:23,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:23,510 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:23,510 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:23,511 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:23,511 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 20:00:23,511 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 20:00:23,511 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:23,511 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:23,511 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:23,511 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:23,512 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:23,512 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:23,512 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:23,512 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:23,513 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf 2023-02-11 20:00:23,513 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf 2023-02-11 20:00:23,514 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:23,514 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:23,517 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region db772e5c38666094ec2e3b8fb5e805c2 columnFamilyName cf 2023-02-11 20:00:23,519 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf 2023-02-11 20:00:23,519 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf 2023-02-11 20:00:23,520 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f1fc9bf98611ec0266597565431cc2a1 columnFamilyName cf 2023-02-11 20:00:23,521 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(310): Store=db772e5c38666094ec2e3b8fb5e805c2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:23,524 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:23,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:23,528 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf 2023-02-11 20:00:23,528 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(310): Store=f1fc9bf98611ec0266597565431cc2a1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:23,528 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf 2023-02-11 20:00:23,529 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a4676c43c8662115dfd922e9ff72451 columnFamilyName cf 2023-02-11 20:00:23,531 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(310): Store=1a4676c43c8662115dfd922e9ff72451/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:23,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:23,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:23,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:23,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:23,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:23,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:23,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:23,547 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 20:00:23,557 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened db772e5c38666094ec2e3b8fb5e805c2; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64157266, jitterRate=-0.043982237577438354}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:23,557 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for db772e5c38666094ec2e3b8fb5e805c2: Running coprocessor pre-open hook at 1676145623502Writing region info on filesystem at 1676145623502Initializing all the Stores at 1676145623506 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145623506Cleaning up temporary data from old regions at 1676145623526 (+20 ms)Cleaning up detritus from prior splits at 1676145623527 (+1 ms)Running coprocessor post-open hooks at 1676145623557 (+30 ms)Region opened successfully at 1676145623557 2023-02-11 20:00:23,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 20:00:23,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1a4676c43c8662115dfd922e9ff72451; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61642297, jitterRate=-0.0814581960439682}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:23,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2., pid=19, masterSystemTime=1676145623493 2023-02-11 20:00:23,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1a4676c43c8662115dfd922e9ff72451: Running coprocessor pre-open hook at 1676145623512Writing region info on filesystem at 1676145623512Initializing all the Stores at 1676145623514 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145623514Cleaning up temporary data from old regions at 1676145623536 (+22 ms)Cleaning up detritus from prior splits at 1676145623536Running coprocessor post-open hooks at 1676145623559 (+23 ms)Region opened successfully at 1676145623559 2023-02-11 20:00:23,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 20:00:23,563 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f1fc9bf98611ec0266597565431cc2a1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66249309, jitterRate=-0.012808367609977722}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:23,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f1fc9bf98611ec0266597565431cc2a1: Running coprocessor pre-open hook at 1676145623512Writing region info on filesystem at 1676145623512Initializing all the Stores at 1676145623514 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145623514Cleaning up temporary data from old regions at 1676145623539 (+25 ms)Cleaning up detritus from prior splits at 1676145623540 (+1 ms)Running coprocessor post-open hooks at 1676145623563 (+23 ms)Region opened successfully at 1676145623563 2023-02-11 20:00:23,563 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451., pid=20, masterSystemTime=1676145623495 2023-02-11 20:00:23,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:23,571 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:23,572 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:23,572 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 20:00:23,572 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1., pid=21, masterSystemTime=1676145623496 2023-02-11 20:00:23,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:23,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:23,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:23,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:23,573 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:23,573 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145623573"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145623573"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145623573"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145623573"}]},"ts":"1676145623573"} 2023-02-11 20:00:23,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:23,576 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:23,576 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:23,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 20:00:23,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:23,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:23,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:23,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:23,580 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:23,584 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:23,585 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145623583"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145623583"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145623583"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145623583"}]},"ts":"1676145623583"} 2023-02-11 20:00:23,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:23,585 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:23,585 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:23,585 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:23,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 20:00:23,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:23,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:23,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:23,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:23,591 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf 2023-02-11 20:00:23,591 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf 2023-02-11 20:00:23,592 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8de911ca888e28adfca434ff163cbba9 columnFamilyName cf 2023-02-11 20:00:23,593 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:23,594 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145623593"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145623593"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145623593"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145623593"}]},"ts":"1676145623593"} 2023-02-11 20:00:23,595 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=10 2023-02-11 20:00:23,604 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf 2023-02-11 20:00:23,602 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=11 2023-02-11 20:00:23,601 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN in 423 msec 2023-02-11 20:00:23,599 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN in 420 msec 2023-02-11 20:00:23,598 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=15 2023-02-11 20:00:23,595 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(310): Store=8de911ca888e28adfca434ff163cbba9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:23,610 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=15, state=SUCCESS; OpenRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924 in 244 msec 2023-02-11 20:00:23,610 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=11, state=SUCCESS; OpenRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887 in 247 msec 2023-02-11 20:00:23,610 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf 2023-02-11 20:00:23,609 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=10, state=SUCCESS; OpenRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841 in 236 msec 2023-02-11 20:00:23,609 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:23,612 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7c040ebb400ff6a8415e55ad85bbbe0e columnFamilyName cf 2023-02-11 20:00:23,614 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:23,617 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:23,617 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN in 435 msec 2023-02-11 20:00:23,618 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(310): Store=7c040ebb400ff6a8415e55ad85bbbe0e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:23,620 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf 2023-02-11 20:00:23,620 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf 2023-02-11 20:00:23,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:23,621 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c630b1709e678e0b00c070c7cac86d30 columnFamilyName cf 2023-02-11 20:00:23,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:23,623 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(310): Store=c630b1709e678e0b00c070c7cac86d30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:23,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:23,626 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:23,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:23,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:23,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 20:00:23,633 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 8de911ca888e28adfca434ff163cbba9; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67083422, jitterRate=-3.7911534309387207E-4}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:23,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 8de911ca888e28adfca434ff163cbba9: Running coprocessor pre-open hook at 1676145623573Writing region info on filesystem at 1676145623573Initializing all the Stores at 1676145623577 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145623577Cleaning up temporary data from old regions at 1676145623619 (+42 ms)Cleaning up detritus from prior splits at 1676145623620 (+1 ms)Running coprocessor post-open hooks at 1676145623633 (+13 ms)Region opened successfully at 1676145623633 2023-02-11 20:00:23,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:23,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 20:00:23,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9., pid=16, masterSystemTime=1676145623493 2023-02-11 20:00:23,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7c040ebb400ff6a8415e55ad85bbbe0e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68052514, jitterRate=0.01406148076057434}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:23,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Running coprocessor pre-open hook at 1676145623577Writing region info on filesystem at 1676145623577Initializing all the Stores at 1676145623579 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145623579Cleaning up temporary data from old regions at 1676145623622 (+43 ms)Cleaning up detritus from prior splits at 1676145623624 (+2 ms)Running coprocessor post-open hooks at 1676145623637 (+13 ms)Region opened successfully at 1676145623637 2023-02-11 20:00:23,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e., pid=17, masterSystemTime=1676145623495 2023-02-11 20:00:23,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:23,643 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:23,642 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:23,641 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:23,645 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:23,645 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145623643"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145623643"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145623643"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145623643"}]},"ts":"1676145623643"} 2023-02-11 20:00:23,645 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145623641"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145623641"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145623641"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145623641"}]},"ts":"1676145623641"} 2023-02-11 20:00:23,645 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:23,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 20:00:23,649 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c630b1709e678e0b00c070c7cac86d30; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74510910, jitterRate=0.11029908061027527}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:23,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c630b1709e678e0b00c070c7cac86d30: Running coprocessor pre-open hook at 1676145623589Writing region info on filesystem at 1676145623589Initializing all the Stores at 1676145623592 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145623592Cleaning up temporary data from old regions at 1676145623627 (+35 ms)Cleaning up detritus from prior splits at 1676145623630 (+3 ms)Running coprocessor post-open hooks at 1676145623649 (+19 ms)Region opened successfully at 1676145623649 2023-02-11 20:00:23,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30., pid=18, masterSystemTime=1676145623496 2023-02-11 20:00:23,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:23,655 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:23,655 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=12 2023-02-11 20:00:23,656 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=12, state=SUCCESS; OpenRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924 in 309 msec 2023-02-11 20:00:23,655 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:23,657 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145623655"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145623655"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145623655"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145623655"}]},"ts":"1676145623655"} 2023-02-11 20:00:23,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-11 20:00:23,658 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=14 2023-02-11 20:00:23,661 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=14, state=SUCCESS; OpenRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841 in 313 msec 2023-02-11 20:00:23,663 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN in 480 msec 2023-02-11 20:00:23,664 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN in 484 msec 2023-02-11 20:00:23,666 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=13 2023-02-11 20:00:23,666 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=13, state=SUCCESS; OpenRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887 in 321 msec 2023-02-11 20:00:23,672 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=9 2023-02-11 20:00:23,672 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN in 491 msec 2023-02-11 20:00:23,673 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676145622835 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 20:00:23,674 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145623673"}]},"ts":"1676145623673"} 2023-02-11 20:00:23,689 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=ENABLED in hbase:meta 2023-02-11 20:00:23,693 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676145622835 execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 20:00:23,696 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1676145622835 in 825 msec 2023-02-11 20:00:24,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-11 20:00:24,411 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1676145622835, procId: 9 completed 2023-02-11 20:00:24,422 DEBUG [Listener at localhost/33691] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676145622835 2023-02-11 20:00:24,423 DEBUG [Listener at localhost/33691] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:24,426 INFO [Listener at localhost/33691] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 20:00:24,452 DEBUG [hconnection-0x7415b278-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:24,456 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45080, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:24,461 DEBUG [hconnection-0x7415b278-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:24,465 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36932, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:24,476 INFO [Listener at localhost/33691] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676145622835 2023-02-11 20:00:24,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676145622835 2023-02-11 20:00:24,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:24,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-11 20:00:24,494 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145624494"}]},"ts":"1676145624494"} 2023-02-11 20:00:24,497 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=DISABLING in hbase:meta 2023-02-11 20:00:24,499 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676145622835 to state=DISABLING 2023-02-11 20:00:24,503 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN}] 2023-02-11 20:00:24,506 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN 2023-02-11 20:00:24,506 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN 2023-02-11 20:00:24,506 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN 2023-02-11 20:00:24,506 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN 2023-02-11 20:00:24,506 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN 2023-02-11 20:00:24,509 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:24,509 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:24,509 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:24,509 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:24,509 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145624509"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145624509"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145624509"}]},"ts":"1676145624509"} 2023-02-11 20:00:24,509 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145624509"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145624509"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145624509"}]},"ts":"1676145624509"} 2023-02-11 20:00:24,509 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145624509"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145624509"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145624509"}]},"ts":"1676145624509"} 2023-02-11 20:00:24,509 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:24,509 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145624509"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145624509"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145624509"}]},"ts":"1676145624509"} 2023-02-11 20:00:24,510 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145624509"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145624509"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145624509"}]},"ts":"1676145624509"} 2023-02-11 20:00:24,512 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=27, state=RUNNABLE; CloseRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:24,513 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=26, state=RUNNABLE; CloseRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:24,521 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=25, state=RUNNABLE; CloseRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:24,523 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=24, state=RUNNABLE; CloseRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:24,525 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=28, state=RUNNABLE; CloseRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:24,530 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN 2023-02-11 20:00:24,532 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:24,532 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145624532"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145624532"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145624532"}]},"ts":"1676145624532"} 2023-02-11 20:00:24,535 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=23, state=RUNNABLE; CloseRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:24,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:24,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:24,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 8de911ca888e28adfca434ff163cbba9, disabling compactions & flushes 2023-02-11 20:00:24,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:24,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:24,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c630b1709e678e0b00c070c7cac86d30, disabling compactions & flushes 2023-02-11 20:00:24,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:24,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:24,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. after waiting 0 ms 2023-02-11 20:00:24,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:24,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:24,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. after waiting 0 ms 2023-02-11 20:00:24,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:24,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1a4676c43c8662115dfd922e9ff72451, disabling compactions & flushes 2023-02-11 20:00:24,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:24,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:24,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. after waiting 0 ms 2023-02-11 20:00:24,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:24,694 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 20:00:24,695 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 20:00:24,695 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:24,695 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 8de911ca888e28adfca434ff163cbba9: Waiting for close lock at 1676145624681Running coprocessor pre-close hooks at 1676145624681Disabling compacts and flushes for region at 1676145624681Disabling writes for close at 1676145624682 (+1 ms)Writing region close event to WAL at 1676145624683 (+1 ms)Running coprocessor post-close hooks at 1676145624695 (+12 ms)Closed at 1676145624695 2023-02-11 20:00:24,696 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:24,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1a4676c43c8662115dfd922e9ff72451: Waiting for close lock at 1676145624682Running coprocessor pre-close hooks at 1676145624682Disabling compacts and flushes for region at 1676145624682Disabling writes for close at 1676145624684 (+2 ms)Writing region close event to WAL at 1676145624685 (+1 ms)Running coprocessor post-close hooks at 1676145624695 (+10 ms)Closed at 1676145624695 2023-02-11 20:00:24,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 20:00:24,701 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:24,702 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c630b1709e678e0b00c070c7cac86d30: Waiting for close lock at 1676145624681Running coprocessor pre-close hooks at 1676145624681Disabling compacts and flushes for region at 1676145624681Disabling writes for close at 1676145624682 (+1 ms)Writing region close event to WAL at 1676145624683 (+1 ms)Running coprocessor post-close hooks at 1676145624701 (+18 ms)Closed at 1676145624701 2023-02-11 20:00:24,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:24,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:24,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7c040ebb400ff6a8415e55ad85bbbe0e, disabling compactions & flushes 2023-02-11 20:00:24,703 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:24,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:24,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. after waiting 0 ms 2023-02-11 20:00:24,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:24,704 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=CLOSED 2023-02-11 20:00:24,705 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145624704"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145624704"}]},"ts":"1676145624704"} 2023-02-11 20:00:24,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:24,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:24,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing db772e5c38666094ec2e3b8fb5e805c2, disabling compactions & flushes 2023-02-11 20:00:24,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:24,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:24,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. after waiting 0 ms 2023-02-11 20:00:24,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:24,706 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=CLOSED 2023-02-11 20:00:24,707 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145624706"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145624706"}]},"ts":"1676145624706"} 2023-02-11 20:00:24,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:24,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:24,708 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=CLOSED 2023-02-11 20:00:24,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f1fc9bf98611ec0266597565431cc2a1, disabling compactions & flushes 2023-02-11 20:00:24,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:24,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:24,709 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145624708"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145624708"}]},"ts":"1676145624708"} 2023-02-11 20:00:24,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. after waiting 0 ms 2023-02-11 20:00:24,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:24,716 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=28 2023-02-11 20:00:24,716 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=28, state=SUCCESS; CloseRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924 in 183 msec 2023-02-11 20:00:24,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 20:00:24,719 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:24,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Waiting for close lock at 1676145624702Running coprocessor pre-close hooks at 1676145624702Disabling compacts and flushes for region at 1676145624702Disabling writes for close at 1676145624703 (+1 ms)Writing region close event to WAL at 1676145624704 (+1 ms)Running coprocessor post-close hooks at 1676145624719 (+15 ms)Closed at 1676145624719 2023-02-11 20:00:24,719 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=27 2023-02-11 20:00:24,719 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=27, state=SUCCESS; CloseRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841 in 202 msec 2023-02-11 20:00:24,720 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN in 215 msec 2023-02-11 20:00:24,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 20:00:24,722 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=26 2023-02-11 20:00:24,722 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=26, state=SUCCESS; CloseRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887 in 204 msec 2023-02-11 20:00:24,722 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:24,722 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:24,723 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for db772e5c38666094ec2e3b8fb5e805c2: Waiting for close lock at 1676145624706Running coprocessor pre-close hooks at 1676145624706Disabling compacts and flushes for region at 1676145624706Disabling writes for close at 1676145624706Writing region close event to WAL at 1676145624709 (+3 ms)Running coprocessor post-close hooks at 1676145624722 (+13 ms)Closed at 1676145624722 2023-02-11 20:00:24,723 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN in 217 msec 2023-02-11 20:00:24,724 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=CLOSED 2023-02-11 20:00:24,724 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145624724"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145624724"}]},"ts":"1676145624724"} 2023-02-11 20:00:24,725 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 20:00:24,726 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN in 220 msec 2023-02-11 20:00:24,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:24,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f1fc9bf98611ec0266597565431cc2a1: Waiting for close lock at 1676145624707Running coprocessor pre-close hooks at 1676145624707Disabling compacts and flushes for region at 1676145624707Disabling writes for close at 1676145624709 (+2 ms)Writing region close event to WAL at 1676145624712 (+3 ms)Running coprocessor post-close hooks at 1676145624726 (+14 ms)Closed at 1676145624726 2023-02-11 20:00:24,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:24,729 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=CLOSED 2023-02-11 20:00:24,729 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145624729"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145624729"}]},"ts":"1676145624729"} 2023-02-11 20:00:24,730 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:24,731 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=CLOSED 2023-02-11 20:00:24,731 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145624731"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145624731"}]},"ts":"1676145624731"} 2023-02-11 20:00:24,733 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=25 2023-02-11 20:00:24,733 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=25, state=SUCCESS; CloseRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924 in 205 msec 2023-02-11 20:00:24,736 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN in 231 msec 2023-02-11 20:00:24,744 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=23 2023-02-11 20:00:24,744 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=23, state=SUCCESS; CloseRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841 in 201 msec 2023-02-11 20:00:24,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-11 20:00:24,747 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=24 2023-02-11 20:00:24,747 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=24, state=SUCCESS; CloseRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887 in 218 msec 2023-02-11 20:00:24,747 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN in 242 msec 2023-02-11 20:00:24,752 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=22 2023-02-11 20:00:24,752 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN in 245 msec 2023-02-11 20:00:24,754 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145624754"}]},"ts":"1676145624754"} 2023-02-11 20:00:24,757 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=DISABLED in hbase:meta 2023-02-11 20:00:24,759 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676145622835 to state=DISABLED 2023-02-11 20:00:24,765 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676145622835 in 278 msec 2023-02-11 20:00:25,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-11 20:00:25,248 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676145622835, procId: 22 completed 2023-02-11 20:00:25,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:25,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676145625259 (current time:1676145625259). 2023-02-11 20:00:25,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 20:00:25,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(139): Connect 0x6977dd15 to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:25,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@82f7956, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:25,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:25,271 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42168, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:25,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6977dd15 to 127.0.0.1:51732 2023-02-11 20:00:25,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:00:25,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 20:00:25,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:25,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:25,294 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:25,295 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE; timeout=600000, timestamp=1676146225295 2023-02-11 20:00:25,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:25,295 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676145622835 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676145622835 2023-02-11 20:00:25,300 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 20:00:25,300 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 20:00:25,320 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676145622835 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } 2023-02-11 20:00:25,323 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:25,323 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:25,323 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:25,323 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:25,324 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:25,323 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:25,323 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:25,328 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:25,328 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:25,328 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:25,328 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:25,328 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:25,328 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:25,328 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:25,335 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:25,335 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:25,335 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:25,335 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:25,335 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:25,335 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:25,336 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:25,551 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 20:00:25,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } is done 2023-02-11 20:00:25,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(404): Snapshoting '{ ss=emptySnaptb-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 }' is still in progress! 2023-02-11 20:00:25,555 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-11 20:00:25,791 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676145622835 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } as finished. 2023-02-11 20:00:25,791 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676145622835 2023-02-11 20:00:25,794 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/emptySnaptb-1676145622835 2023-02-11 20:00:25,838 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/emptySnaptb-1676145622835 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/emptySnaptb-1676145622835 2023-02-11 20:00:25,870 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676145622835 of table testRestoreSnapshot-1676145622835 completed 2023-02-11 20:00:25,870 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/emptySnaptb-1676145622835 2023-02-11 20:00:25,871 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/emptySnaptb-1676145622835 2023-02-11 20:00:25,871 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676145622835 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676145622835 at 1676145625295Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676145622835 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } at 1676145625320 (+25 ms)Consolidate snapshot: emptySnaptb-1676145622835 at 1676145625791 (+471 ms)Loading Region manifests for emptySnaptb-1676145622835 at 1676145625791Writing data manifest for emptySnaptb-1676145622835 at 1676145625809 (+18 ms)Verifying snapshot: emptySnaptb-1676145622835 at 1676145625825 (+16 ms)Snapshot emptySnaptb-1676145622835 of table testRestoreSnapshot-1676145622835 completed at 1676145625870 (+45 ms) 2023-02-11 20:00:25,872 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:25,874 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE in 586 msec 2023-02-11 20:00:26,055 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 20:00:26,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } is done 2023-02-11 20:00:26,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 20:00:26,059 INFO [Listener at localhost/33691] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676145622835 2023-02-11 20:00:26,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676145622835 2023-02-11 20:00:26,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:26,074 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676145622835 2023-02-11 20:00:26,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-11 20:00:26,074 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145626074"}]},"ts":"1676145626074"} 2023-02-11 20:00:26,076 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=ENABLING in hbase:meta 2023-02-11 20:00:26,080 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 20:00:26,082 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN}] 2023-02-11 20:00:26,085 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN 2023-02-11 20:00:26,085 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN 2023-02-11 20:00:26,085 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN 2023-02-11 20:00:26,085 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN 2023-02-11 20:00:26,086 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN 2023-02-11 20:00:26,086 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:26,086 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:26,086 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44175,1676145619924; forceNewPlan=false, retain=true 2023-02-11 20:00:26,087 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42011,1676145619841; forceNewPlan=false, retain=true 2023-02-11 20:00:26,087 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:26,087 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44175,1676145619924; forceNewPlan=false, retain=true 2023-02-11 20:00:26,087 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:26,087 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35937,1676145619887; forceNewPlan=false, retain=true 2023-02-11 20:00:26,088 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:26,088 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35937,1676145619887; forceNewPlan=false, retain=true 2023-02-11 20:00:26,089 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN 2023-02-11 20:00:26,090 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:26,090 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42011,1676145619841; forceNewPlan=false, retain=true 2023-02-11 20:00:26,237 INFO [jenkins-hbase4:39971] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 20:00:26,238 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:26,238 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:26,238 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:26,238 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:26,238 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:26,238 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145626238"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145626238"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145626238"}]},"ts":"1676145626238"} 2023-02-11 20:00:26,238 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145626238"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145626238"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145626238"}]},"ts":"1676145626238"} 2023-02-11 20:00:26,238 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145626238"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145626238"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145626238"}]},"ts":"1676145626238"} 2023-02-11 20:00:26,238 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145626238"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145626238"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145626238"}]},"ts":"1676145626238"} 2023-02-11 20:00:26,238 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145626238"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145626238"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145626238"}]},"ts":"1676145626238"} 2023-02-11 20:00:26,241 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=42, state=RUNNABLE; OpenRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:26,242 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=39, state=RUNNABLE; OpenRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:26,243 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=41, state=RUNNABLE; OpenRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:26,245 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=37, state=RUNNABLE; OpenRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:26,246 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=40, state=RUNNABLE; OpenRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:26,248 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:26,248 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145626248"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145626248"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145626248"}]},"ts":"1676145626248"} 2023-02-11 20:00:26,250 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=38, state=RUNNABLE; OpenRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:26,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-11 20:00:26,398 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:26,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 20:00:26,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:26,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:26,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:26,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:26,407 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:26,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 20:00:26,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:26,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:26,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:26,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:26,409 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:26,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 20:00:26,409 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:26,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:26,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:26,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:26,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:26,411 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf 2023-02-11 20:00:26,411 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf 2023-02-11 20:00:26,411 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a4676c43c8662115dfd922e9ff72451 columnFamilyName cf 2023-02-11 20:00:26,413 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(310): Store=1a4676c43c8662115dfd922e9ff72451/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:26,415 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:26,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:26,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:26,419 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:26,420 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf 2023-02-11 20:00:26,420 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf 2023-02-11 20:00:26,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:26,421 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf 2023-02-11 20:00:26,421 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf 2023-02-11 20:00:26,421 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8de911ca888e28adfca434ff163cbba9 columnFamilyName cf 2023-02-11 20:00:26,422 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1a4676c43c8662115dfd922e9ff72451; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73952145, jitterRate=0.10197283327579498}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:26,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1a4676c43c8662115dfd922e9ff72451: Running coprocessor pre-open hook at 1676145626399Writing region info on filesystem at 1676145626399Initializing all the Stores at 1676145626406 (+7 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145626406Cleaning up temporary data from old regions at 1676145626417 (+11 ms)Cleaning up detritus from prior splits at 1676145626417Running coprocessor post-open hooks at 1676145626422 (+5 ms)Region opened successfully at 1676145626422 2023-02-11 20:00:26,422 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c630b1709e678e0b00c070c7cac86d30 columnFamilyName cf 2023-02-11 20:00:26,423 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451., pid=43, masterSystemTime=1676145626393 2023-02-11 20:00:26,423 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(310): Store=8de911ca888e28adfca434ff163cbba9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:26,425 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:26,425 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(310): Store=c630b1709e678e0b00c070c7cac86d30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:26,425 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:26,425 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:26,425 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:26,425 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 20:00:26,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:26,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:26,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:26,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:26,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:26,427 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:26,427 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145626426"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145626426"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145626426"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145626426"}]},"ts":"1676145626426"} 2023-02-11 20:00:26,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:26,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:26,432 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:26,432 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=42 2023-02-11 20:00:26,432 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=42, state=SUCCESS; OpenRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924 in 188 msec 2023-02-11 20:00:26,433 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:26,433 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:26,436 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf 2023-02-11 20:00:26,436 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf 2023-02-11 20:00:26,436 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 8de911ca888e28adfca434ff163cbba9; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75187249, jitterRate=0.120377317070961}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:26,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 8de911ca888e28adfca434ff163cbba9: Running coprocessor pre-open hook at 1676145626409Writing region info on filesystem at 1676145626409Initializing all the Stores at 1676145626410 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145626410Cleaning up temporary data from old regions at 1676145626428 (+18 ms)Cleaning up detritus from prior splits at 1676145626429 (+1 ms)Running coprocessor post-open hooks at 1676145626436 (+7 ms)Region opened successfully at 1676145626436 2023-02-11 20:00:26,436 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7c040ebb400ff6a8415e55ad85bbbe0e columnFamilyName cf 2023-02-11 20:00:26,436 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN in 352 msec 2023-02-11 20:00:26,437 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c630b1709e678e0b00c070c7cac86d30; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62103245, jitterRate=-0.07458953559398651}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:26,437 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c630b1709e678e0b00c070c7cac86d30: Running coprocessor pre-open hook at 1676145626410Writing region info on filesystem at 1676145626410Initializing all the Stores at 1676145626411 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145626411Cleaning up temporary data from old regions at 1676145626429 (+18 ms)Cleaning up detritus from prior splits at 1676145626429Running coprocessor post-open hooks at 1676145626437 (+8 ms)Region opened successfully at 1676145626437 2023-02-11 20:00:26,438 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30., pid=47, masterSystemTime=1676145626399 2023-02-11 20:00:26,439 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(310): Store=7c040ebb400ff6a8415e55ad85bbbe0e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:26,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:26,440 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:26,440 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:26,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 20:00:26,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:26,440 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:26,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:26,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:26,440 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145626440"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145626440"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145626440"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145626440"}]},"ts":"1676145626440"} 2023-02-11 20:00:26,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:26,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:26,442 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:26,444 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9., pid=45, masterSystemTime=1676145626398 2023-02-11 20:00:26,446 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=40 2023-02-11 20:00:26,446 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=40, state=SUCCESS; OpenRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887 in 196 msec 2023-02-11 20:00:26,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:26,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:26,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:26,447 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:26,447 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:26,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 20:00:26,447 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:26,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:26,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:26,447 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145626447"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145626447"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145626447"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145626447"}]},"ts":"1676145626447"} 2023-02-11 20:00:26,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:26,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:26,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7c040ebb400ff6a8415e55ad85bbbe0e; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71791914, jitterRate=0.06978288292884827}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:26,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Running coprocessor pre-open hook at 1676145626426Writing region info on filesystem at 1676145626426Initializing all the Stores at 1676145626427 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145626427Cleaning up temporary data from old regions at 1676145626443 (+16 ms)Cleaning up detritus from prior splits at 1676145626443Running coprocessor post-open hooks at 1676145626448 (+5 ms)Region opened successfully at 1676145626448 2023-02-11 20:00:26,449 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN in 365 msec 2023-02-11 20:00:26,450 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e., pid=44, masterSystemTime=1676145626393 2023-02-11 20:00:26,450 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf 2023-02-11 20:00:26,450 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf 2023-02-11 20:00:26,451 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f1fc9bf98611ec0266597565431cc2a1 columnFamilyName cf 2023-02-11 20:00:26,453 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=41 2023-02-11 20:00:26,453 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=41, state=SUCCESS; OpenRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841 in 207 msec 2023-02-11 20:00:26,453 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(310): Store=f1fc9bf98611ec0266597565431cc2a1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:26,453 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:26,453 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:26,454 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:26,454 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145626454"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145626454"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145626454"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145626454"}]},"ts":"1676145626454"} 2023-02-11 20:00:26,456 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN in 372 msec 2023-02-11 20:00:26,457 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:26,457 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:26,459 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=39 2023-02-11 20:00:26,459 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=39, state=SUCCESS; OpenRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924 in 214 msec 2023-02-11 20:00:26,459 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:26,461 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN in 378 msec 2023-02-11 20:00:26,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:26,464 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f1fc9bf98611ec0266597565431cc2a1; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72357988, jitterRate=0.07821804285049438}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:26,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f1fc9bf98611ec0266597565431cc2a1: Running coprocessor pre-open hook at 1676145626441Writing region info on filesystem at 1676145626441Initializing all the Stores at 1676145626442 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145626442Cleaning up temporary data from old regions at 1676145626460 (+18 ms)Cleaning up detritus from prior splits at 1676145626460Running coprocessor post-open hooks at 1676145626464 (+4 ms)Region opened successfully at 1676145626464 2023-02-11 20:00:26,465 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1., pid=48, masterSystemTime=1676145626399 2023-02-11 20:00:26,467 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf 2023-02-11 20:00:26,467 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf 2023-02-11 20:00:26,467 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:26,467 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:26,467 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region db772e5c38666094ec2e3b8fb5e805c2 columnFamilyName cf 2023-02-11 20:00:26,468 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:26,468 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145626467"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145626467"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145626467"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145626467"}]},"ts":"1676145626467"} 2023-02-11 20:00:26,470 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(310): Store=db772e5c38666094ec2e3b8fb5e805c2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:26,471 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:26,472 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=38 2023-02-11 20:00:26,472 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=38, state=SUCCESS; OpenRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887 in 220 msec 2023-02-11 20:00:26,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:26,474 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN in 391 msec 2023-02-11 20:00:26,477 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:26,478 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened db772e5c38666094ec2e3b8fb5e805c2; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75410757, jitterRate=0.1237078458070755}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:26,478 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for db772e5c38666094ec2e3b8fb5e805c2: Running coprocessor pre-open hook at 1676145626448Writing region info on filesystem at 1676145626448Initializing all the Stores at 1676145626450 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145626450Cleaning up temporary data from old regions at 1676145626474 (+24 ms)Cleaning up detritus from prior splits at 1676145626474Running coprocessor post-open hooks at 1676145626478 (+4 ms)Region opened successfully at 1676145626478 2023-02-11 20:00:26,479 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2., pid=46, masterSystemTime=1676145626398 2023-02-11 20:00:26,481 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:26,481 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:26,482 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:26,482 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145626481"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145626481"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145626481"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145626481"}]},"ts":"1676145626481"} 2023-02-11 20:00:26,486 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=37 2023-02-11 20:00:26,486 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=37, state=SUCCESS; OpenRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841 in 239 msec 2023-02-11 20:00:26,488 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-02-11 20:00:26,489 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN in 405 msec 2023-02-11 20:00:26,490 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145626489"}]},"ts":"1676145626489"} 2023-02-11 20:00:26,491 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=ENABLED in hbase:meta 2023-02-11 20:00:26,493 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676145622835' was successfully enabled. 2023-02-11 20:00:26,500 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676145622835 in 429 msec 2023-02-11 20:00:26,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-11 20:00:26,827 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676145622835, procId: 36 completed 2023-02-11 20:00:26,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:26,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:26,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:26,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:26,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:26,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:26,886 DEBUG [Listener at localhost/33691] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676145622835 2023-02-11 20:00:26,887 DEBUG [Listener at localhost/33691] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:26,888 INFO [Listener at localhost/33691] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 20:00:26,914 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 20:00:26,917 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 20:00:26,920 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 20:00:26,922 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 20:00:26,923 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 20:00:26,934 INFO [Listener at localhost/33691] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676145622835 2023-02-11 20:00:26,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676145622835 2023-02-11 20:00:26,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:26,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-11 20:00:26,941 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145626941"}]},"ts":"1676145626941"} 2023-02-11 20:00:26,943 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=DISABLING in hbase:meta 2023-02-11 20:00:26,944 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676145622835 to state=DISABLING 2023-02-11 20:00:26,946 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN}] 2023-02-11 20:00:26,948 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN 2023-02-11 20:00:26,949 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN 2023-02-11 20:00:26,949 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN 2023-02-11 20:00:26,949 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN 2023-02-11 20:00:26,949 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN 2023-02-11 20:00:26,950 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:26,950 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:26,951 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145626950"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145626950"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145626950"}]},"ts":"1676145626950"} 2023-02-11 20:00:26,951 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145626950"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145626950"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145626950"}]},"ts":"1676145626950"} 2023-02-11 20:00:26,951 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:26,951 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145626951"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145626951"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145626951"}]},"ts":"1676145626951"} 2023-02-11 20:00:26,953 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:26,953 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:26,953 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145626952"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145626952"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145626952"}]},"ts":"1676145626952"} 2023-02-11 20:00:26,953 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145626953"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145626953"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145626953"}]},"ts":"1676145626953"} 2023-02-11 20:00:26,953 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=50, state=RUNNABLE; CloseRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:26,954 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=51, state=RUNNABLE; CloseRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:26,956 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=53, state=RUNNABLE; CloseRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:26,961 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=55, state=RUNNABLE; CloseRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:26,962 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=54, state=RUNNABLE; CloseRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:26,969 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN 2023-02-11 20:00:26,970 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:26,970 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145626970"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145626970"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145626970"}]},"ts":"1676145626970"} 2023-02-11 20:00:26,973 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=52, state=RUNNABLE; CloseRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:27,109 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:27,109 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:27,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing db772e5c38666094ec2e3b8fb5e805c2, disabling compactions & flushes 2023-02-11 20:00:27,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c630b1709e678e0b00c070c7cac86d30, disabling compactions & flushes 2023-02-11 20:00:27,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:27,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:27,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:27,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:27,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. after waiting 0 ms 2023-02-11 20:00:27,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. after waiting 0 ms 2023-02-11 20:00:27,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:27,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:27,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing db772e5c38666094ec2e3b8fb5e805c2 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-11 20:00:27,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c630b1709e678e0b00c070c7cac86d30 1/1 column families, dataSize=1.24 KB heapSize=2.92 KB 2023-02-11 20:00:27,121 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:27,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1a4676c43c8662115dfd922e9ff72451, disabling compactions & flushes 2023-02-11 20:00:27,122 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:27,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:27,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. after waiting 0 ms 2023-02-11 20:00:27,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:27,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1a4676c43c8662115dfd922e9ff72451 1/1 column families, dataSize=23.35 KB heapSize=50.45 KB 2023-02-11 20:00:27,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-11 20:00:27,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f 2023-02-11 20:00:27,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 2023-02-11 20:00:27,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 2023-02-11 20:00:27,274 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 20:00:27,348 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/.tmp/cf/e793845eb1f447169fd725e5a836da1b 2023-02-11 20:00:27,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/.tmp/cf/e793845eb1f447169fd725e5a836da1b as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/e793845eb1f447169fd725e5a836da1b 2023-02-11 20:00:27,380 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=23.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/.tmp/cf/0bcc1b49b65f42d389fcb662a909dbf1 2023-02-11 20:00:27,383 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/e793845eb1f447169fd725e5a836da1b, entries=26, sequenceid=8, filesize=9.3 K 2023-02-11 20:00:27,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/.tmp/cf/cf1d617c62bd452786cb06efe103dd00 2023-02-11 20:00:27,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for db772e5c38666094ec2e3b8fb5e805c2 in 276ms, sequenceid=8, compaction requested=false 2023-02-11 20:00:27,390 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676145622835' 2023-02-11 20:00:27,391 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/.tmp/cf/0bcc1b49b65f42d389fcb662a909dbf1 as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/0bcc1b49b65f42d389fcb662a909dbf1 2023-02-11 20:00:27,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/.tmp/cf/cf1d617c62bd452786cb06efe103dd00 as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/cf1d617c62bd452786cb06efe103dd00 2023-02-11 20:00:27,402 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 20:00:27,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/0bcc1b49b65f42d389fcb662a909dbf1, entries=357, sequenceid=8, filesize=65.7 K 2023-02-11 20:00:27,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:27,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for db772e5c38666094ec2e3b8fb5e805c2: Waiting for close lock at 1676145627109Running coprocessor pre-close hooks at 1676145627109Disabling compacts and flushes for region at 1676145627109Disabling writes for close at 1676145627111 (+2 ms)Obtaining lock to block concurrent updates at 1676145627111Preparing flush snapshotting stores in db772e5c38666094ec2e3b8fb5e805c2 at 1676145627111Finished memstore snapshotting testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2., syncing WAL and waiting on mvcc, flushsize=dataSize=1740, getHeapSize=3984, getOffHeapSize=0, getCellsCount=26 at 1676145627121 (+10 ms)Flushing stores of testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. at 1676145627123 (+2 ms)Flushing db772e5c38666094ec2e3b8fb5e805c2/cf: creating writer at 1676145627126 (+3 ms)Flushing mob file db772e5c38666094ec2e3b8fb5e805c2/cf: appending metadata at 1676145627172 (+46 ms)Flushing mob file db772e5c38666094ec2e3b8fb5e805c2/cf: closing flushed file at 1676145627172Flushing db772e5c38666094ec2e3b8fb5e805c2/cf: appending metadata at 1676145627271 (+99 ms)Flushing db772e5c38666094ec2e3b8fb5e805c2/cf: closing flushed file at 1676145627276 (+5 ms)Flushing db772e5c38666094ec2e3b8fb5e805c2/cf: reopening flushed file at 1676145627367 (+91 ms)Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for db772e5c38666094ec2e3b8fb5e805c2 in 276ms, sequenceid=8, compaction requested=false at 1676145627388 (+21 ms)Writing region close event to WAL at 1676145627396 (+8 ms)Running coprocessor post-close hooks at 1676145627405 (+9 ms)Closed at 1676145627405 2023-02-11 20:00:27,406 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~23.35 KB/23915, heapSize ~50.44 KB/51648, currentSize=0 B/0 for 1a4676c43c8662115dfd922e9ff72451 in 284ms, sequenceid=8, compaction requested=false 2023-02-11 20:00:27,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:27,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:27,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 8de911ca888e28adfca434ff163cbba9, disabling compactions & flushes 2023-02-11 20:00:27,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:27,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:27,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. after waiting 0 ms 2023-02-11 20:00:27,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:27,409 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 8de911ca888e28adfca434ff163cbba9 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-11 20:00:27,411 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=CLOSED 2023-02-11 20:00:27,411 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145627411"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145627411"}]},"ts":"1676145627411"} 2023-02-11 20:00:27,413 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/cf1d617c62bd452786cb06efe103dd00, entries=19, sequenceid=8, filesize=8.1 K 2023-02-11 20:00:27,416 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.24 KB/1271, heapSize ~2.91 KB/2976, currentSize=0 B/0 for c630b1709e678e0b00c070c7cac86d30 in 305ms, sequenceid=8, compaction requested=false 2023-02-11 20:00:27,418 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=50 2023-02-11 20:00:27,418 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=50, state=SUCCESS; CloseRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841 in 460 msec 2023-02-11 20:00:27,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 20:00:27,421 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:27,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1a4676c43c8662115dfd922e9ff72451: Waiting for close lock at 1676145627121Running coprocessor pre-close hooks at 1676145627121Disabling compacts and flushes for region at 1676145627121Disabling writes for close at 1676145627122 (+1 ms)Obtaining lock to block concurrent updates at 1676145627123 (+1 ms)Preparing flush snapshotting stores in 1a4676c43c8662115dfd922e9ff72451 at 1676145627123Finished memstore snapshotting testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451., syncing WAL and waiting on mvcc, flushsize=dataSize=23915, getHeapSize=51648, getOffHeapSize=0, getCellsCount=357 at 1676145627123Flushing stores of testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. at 1676145627123Flushing 1a4676c43c8662115dfd922e9ff72451/cf: creating writer at 1676145627126 (+3 ms)Flushing mob file 1a4676c43c8662115dfd922e9ff72451/cf: appending metadata at 1676145627188 (+62 ms)Flushing mob file 1a4676c43c8662115dfd922e9ff72451/cf: closing flushed file at 1676145627188Flushing 1a4676c43c8662115dfd922e9ff72451/cf: appending metadata at 1676145627271 (+83 ms)Flushing 1a4676c43c8662115dfd922e9ff72451/cf: closing flushed file at 1676145627283 (+12 ms)Flushing 1a4676c43c8662115dfd922e9ff72451/cf: reopening flushed file at 1676145627392 (+109 ms)Finished flush of dataSize ~23.35 KB/23915, heapSize ~50.44 KB/51648, currentSize=0 B/0 for 1a4676c43c8662115dfd922e9ff72451 in 284ms, sequenceid=8, compaction requested=false at 1676145627406 (+14 ms)Writing region close event to WAL at 1676145627411 (+5 ms)Running coprocessor post-close hooks at 1676145627420 (+9 ms)Closed at 1676145627421 (+1 ms) 2023-02-11 20:00:27,423 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN in 472 msec 2023-02-11 20:00:27,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:27,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:27,428 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=CLOSED 2023-02-11 20:00:27,428 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145627428"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145627428"}]},"ts":"1676145627428"} 2023-02-11 20:00:27,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7c040ebb400ff6a8415e55ad85bbbe0e, disabling compactions & flushes 2023-02-11 20:00:27,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:27,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:27,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. after waiting 0 ms 2023-02-11 20:00:27,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:27,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7c040ebb400ff6a8415e55ad85bbbe0e 1/1 column families, dataSize=2.48 KB heapSize=5.59 KB 2023-02-11 20:00:27,438 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=55 2023-02-11 20:00:27,438 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=55, state=SUCCESS; CloseRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924 in 473 msec 2023-02-11 20:00:27,441 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN in 492 msec 2023-02-11 20:00:27,443 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-11 20:00:27,443 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-11 20:00:27,445 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 20:00:27,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:27,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c630b1709e678e0b00c070c7cac86d30: Waiting for close lock at 1676145627109Running coprocessor pre-close hooks at 1676145627109Disabling compacts and flushes for region at 1676145627109Disabling writes for close at 1676145627111 (+2 ms)Obtaining lock to block concurrent updates at 1676145627111Preparing flush snapshotting stores in c630b1709e678e0b00c070c7cac86d30 at 1676145627112 (+1 ms)Finished memstore snapshotting testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30., syncing WAL and waiting on mvcc, flushsize=dataSize=1271, getHeapSize=2976, getOffHeapSize=0, getCellsCount=19 at 1676145627121 (+9 ms)Flushing stores of testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. at 1676145627123 (+2 ms)Flushing c630b1709e678e0b00c070c7cac86d30/cf: creating writer at 1676145627126 (+3 ms)Flushing mob file c630b1709e678e0b00c070c7cac86d30/cf: appending metadata at 1676145627171 (+45 ms)Flushing mob file c630b1709e678e0b00c070c7cac86d30/cf: closing flushed file at 1676145627171Flushing c630b1709e678e0b00c070c7cac86d30/cf: appending metadata at 1676145627283 (+112 ms)Flushing c630b1709e678e0b00c070c7cac86d30/cf: closing flushed file at 1676145627283Flushing c630b1709e678e0b00c070c7cac86d30/cf: reopening flushed file at 1676145627401 (+118 ms)Finished flush of dataSize ~1.24 KB/1271, heapSize ~2.91 KB/2976, currentSize=0 B/0 for c630b1709e678e0b00c070c7cac86d30 in 305ms, sequenceid=8, compaction requested=false at 1676145627416 (+15 ms)Writing region close event to WAL at 1676145627428 (+12 ms)Running coprocessor post-close hooks at 1676145627446 (+18 ms)Closed at 1676145627447 (+1 ms) 2023-02-11 20:00:27,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:27,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:27,464 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=CLOSED 2023-02-11 20:00:27,464 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145627464"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145627464"}]},"ts":"1676145627464"} 2023-02-11 20:00:27,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f1fc9bf98611ec0266597565431cc2a1, disabling compactions & flushes 2023-02-11 20:00:27,467 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:27,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:27,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. after waiting 0 ms 2023-02-11 20:00:27,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:27,468 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing f1fc9bf98611ec0266597565431cc2a1 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-11 20:00:27,472 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=53 2023-02-11 20:00:27,472 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=53, state=SUCCESS; CloseRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887 in 510 msec 2023-02-11 20:00:27,476 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN in 526 msec 2023-02-11 20:00:27,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 2023-02-11 20:00:27,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 2023-02-11 20:00:27,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 2023-02-11 20:00:27,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/.tmp/cf/d185ce9140d04e959efc0b58ace714af 2023-02-11 20:00:27,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/.tmp/cf/d185ce9140d04e959efc0b58ace714af as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/d185ce9140d04e959efc0b58ace714af 2023-02-11 20:00:27,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/.tmp/cf/2e4307f987394952952cf7c59270b4e7 2023-02-11 20:00:27,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/d185ce9140d04e959efc0b58ace714af, entries=29, sequenceid=8, filesize=9.8 K 2023-02-11 20:00:27,539 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/.tmp/cf/f1723f83b8234b60b4690710a470498b 2023-02-11 20:00:27,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/.tmp/cf/2e4307f987394952952cf7c59270b4e7 as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/2e4307f987394952952cf7c59270b4e7 2023-02-11 20:00:27,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 8de911ca888e28adfca434ff163cbba9 in 134ms, sequenceid=8, compaction requested=false 2023-02-11 20:00:27,549 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 20:00:27,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:27,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/2e4307f987394952952cf7c59270b4e7, entries=38, sequenceid=8, filesize=11.4 K 2023-02-11 20:00:27,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 8de911ca888e28adfca434ff163cbba9: Waiting for close lock at 1676145627408Running coprocessor pre-close hooks at 1676145627408Disabling compacts and flushes for region at 1676145627408Disabling writes for close at 1676145627408Obtaining lock to block concurrent updates at 1676145627409 (+1 ms)Preparing flush snapshotting stores in 8de911ca888e28adfca434ff163cbba9 at 1676145627409Finished memstore snapshotting testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9., syncing WAL and waiting on mvcc, flushsize=dataSize=1941, getHeapSize=4416, getOffHeapSize=0, getCellsCount=29 at 1676145627409Flushing stores of testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. at 1676145627411 (+2 ms)Flushing 8de911ca888e28adfca434ff163cbba9/cf: creating writer at 1676145627411Flushing mob file 8de911ca888e28adfca434ff163cbba9/cf: appending metadata at 1676145627429 (+18 ms)Flushing mob file 8de911ca888e28adfca434ff163cbba9/cf: closing flushed file at 1676145627429Flushing 8de911ca888e28adfca434ff163cbba9/cf: appending metadata at 1676145627484 (+55 ms)Flushing 8de911ca888e28adfca434ff163cbba9/cf: closing flushed file at 1676145627484Flushing 8de911ca888e28adfca434ff163cbba9/cf: reopening flushed file at 1676145627527 (+43 ms)Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 8de911ca888e28adfca434ff163cbba9 in 134ms, sequenceid=8, compaction requested=false at 1676145627542 (+15 ms)Writing region close event to WAL at 1676145627544 (+2 ms)Running coprocessor post-close hooks at 1676145627550 (+6 ms)Closed at 1676145627551 (+1 ms) 2023-02-11 20:00:27,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for 7c040ebb400ff6a8415e55ad85bbbe0e in 121ms, sequenceid=8, compaction requested=false 2023-02-11 20:00:27,553 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:27,554 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=CLOSED 2023-02-11 20:00:27,554 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145627554"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145627554"}]},"ts":"1676145627554"} 2023-02-11 20:00:27,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/.tmp/cf/f1723f83b8234b60b4690710a470498b as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/f1723f83b8234b60b4690710a470498b 2023-02-11 20:00:27,560 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=54 2023-02-11 20:00:27,560 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=54, state=SUCCESS; CloseRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841 in 594 msec 2023-02-11 20:00:27,562 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN in 614 msec 2023-02-11 20:00:27,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/f1723f83b8234b60b4690710a470498b, entries=31, sequenceid=8, filesize=10.2 K 2023-02-11 20:00:27,563 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for f1fc9bf98611ec0266597565431cc2a1 in 96ms, sequenceid=8, compaction requested=false 2023-02-11 20:00:27,576 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 20:00:27,577 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:27,577 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f1fc9bf98611ec0266597565431cc2a1: Waiting for close lock at 1676145627462Running coprocessor pre-close hooks at 1676145627462Disabling compacts and flushes for region at 1676145627462Disabling writes for close at 1676145627467 (+5 ms)Obtaining lock to block concurrent updates at 1676145627468 (+1 ms)Preparing flush snapshotting stores in f1fc9bf98611ec0266597565431cc2a1 at 1676145627468Finished memstore snapshotting testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676145627468Flushing stores of testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. at 1676145627469 (+1 ms)Flushing f1fc9bf98611ec0266597565431cc2a1/cf: creating writer at 1676145627470 (+1 ms)Flushing mob file f1fc9bf98611ec0266597565431cc2a1/cf: appending metadata at 1676145627478 (+8 ms)Flushing mob file f1fc9bf98611ec0266597565431cc2a1/cf: closing flushed file at 1676145627478Flushing f1fc9bf98611ec0266597565431cc2a1/cf: appending metadata at 1676145627515 (+37 ms)Flushing f1fc9bf98611ec0266597565431cc2a1/cf: closing flushed file at 1676145627515Flushing f1fc9bf98611ec0266597565431cc2a1/cf: reopening flushed file at 1676145627555 (+40 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for f1fc9bf98611ec0266597565431cc2a1 in 96ms, sequenceid=8, compaction requested=false at 1676145627563 (+8 ms)Writing region close event to WAL at 1676145627572 (+9 ms)Running coprocessor post-close hooks at 1676145627576 (+4 ms)Closed at 1676145627577 (+1 ms) 2023-02-11 20:00:27,578 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 20:00:27,579 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:27,580 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Waiting for close lock at 1676145627427Running coprocessor pre-close hooks at 1676145627427Disabling compacts and flushes for region at 1676145627427Disabling writes for close at 1676145627431 (+4 ms)Obtaining lock to block concurrent updates at 1676145627431Preparing flush snapshotting stores in 7c040ebb400ff6a8415e55ad85bbbe0e at 1676145627431Finished memstore snapshotting testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e., syncing WAL and waiting on mvcc, flushsize=dataSize=2544, getHeapSize=5712, getOffHeapSize=0, getCellsCount=38 at 1676145627431Flushing stores of testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. at 1676145627432 (+1 ms)Flushing 7c040ebb400ff6a8415e55ad85bbbe0e/cf: creating writer at 1676145627433 (+1 ms)Flushing mob file 7c040ebb400ff6a8415e55ad85bbbe0e/cf: appending metadata at 1676145627448 (+15 ms)Flushing mob file 7c040ebb400ff6a8415e55ad85bbbe0e/cf: closing flushed file at 1676145627448Flushing 7c040ebb400ff6a8415e55ad85bbbe0e/cf: appending metadata at 1676145627507 (+59 ms)Flushing 7c040ebb400ff6a8415e55ad85bbbe0e/cf: closing flushed file at 1676145627507Flushing 7c040ebb400ff6a8415e55ad85bbbe0e/cf: reopening flushed file at 1676145627542 (+35 ms)Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for 7c040ebb400ff6a8415e55ad85bbbe0e in 121ms, sequenceid=8, compaction requested=false at 1676145627552 (+10 ms)Writing region close event to WAL at 1676145627570 (+18 ms)Running coprocessor post-close hooks at 1676145627579 (+9 ms)Closed at 1676145627579 2023-02-11 20:00:27,580 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:27,580 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=CLOSED 2023-02-11 20:00:27,581 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145627580"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145627580"}]},"ts":"1676145627580"} 2023-02-11 20:00:27,582 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:27,583 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=CLOSED 2023-02-11 20:00:27,583 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145627583"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145627583"}]},"ts":"1676145627583"} 2023-02-11 20:00:27,586 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=51 2023-02-11 20:00:27,586 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=51, state=SUCCESS; CloseRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887 in 629 msec 2023-02-11 20:00:27,589 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=52 2023-02-11 20:00:27,589 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=52, state=SUCCESS; CloseRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924 in 612 msec 2023-02-11 20:00:27,589 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN in 640 msec 2023-02-11 20:00:27,591 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=49 2023-02-11 20:00:27,591 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN in 643 msec 2023-02-11 20:00:27,593 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145627593"}]},"ts":"1676145627593"} 2023-02-11 20:00:27,595 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=DISABLED in hbase:meta 2023-02-11 20:00:27,597 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676145622835 to state=DISABLED 2023-02-11 20:00:27,600 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676145622835 in 663 msec 2023-02-11 20:00:27,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-11 20:00:27,694 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676145622835, procId: 49 completed 2023-02-11 20:00:27,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:27,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676145627695 (current time:1676145627695). 2023-02-11 20:00:27,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 20:00:27,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(139): Connect 0x2e1eb61b to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:27,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40443ebf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:27,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:27,704 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42176, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:27,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2e1eb61b to 127.0.0.1:51732 2023-02-11 20:00:27,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:00:27,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 20:00:27,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:27,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:27,710 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:27,711 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE; timeout=600000, timestamp=1676146227710 2023-02-11 20:00:27,711 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676145622835 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676145622835 2023-02-11 20:00:27,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:27,711 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 20:00:27,711 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 20:00:27,729 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676145622835 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } 2023-02-11 20:00:27,734 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:27,735 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:27,735 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:27,735 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:27,735 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:27,735 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:27,736 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:27,736 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:27,736 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:27,736 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:27,736 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:27,736 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:27,737 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:27,737 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:27,738 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/0bcc1b49b65f42d389fcb662a909dbf1] hfiles 2023-02-11 20:00:27,738 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/0bcc1b49b65f42d389fcb662a909dbf1 2023-02-11 20:00:27,741 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/2e4307f987394952952cf7c59270b4e7] hfiles 2023-02-11 20:00:27,741 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/2e4307f987394952952cf7c59270b4e7 2023-02-11 20:00:27,741 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/d185ce9140d04e959efc0b58ace714af] hfiles 2023-02-11 20:00:27,741 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/d185ce9140d04e959efc0b58ace714af 2023-02-11 20:00:27,742 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/e793845eb1f447169fd725e5a836da1b] hfiles 2023-02-11 20:00:27,742 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/e793845eb1f447169fd725e5a836da1b 2023-02-11 20:00:27,742 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/cf1d617c62bd452786cb06efe103dd00] hfiles 2023-02-11 20:00:27,742 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/cf1d617c62bd452786cb06efe103dd00 2023-02-11 20:00:27,742 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/f1723f83b8234b60b4690710a470498b] hfiles 2023-02-11 20:00:27,742 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/f1723f83b8234b60b4690710a470498b 2023-02-11 20:00:27,744 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47] hfiles 2023-02-11 20:00:27,744 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 2023-02-11 20:00:27,744 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 2023-02-11 20:00:27,744 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 2023-02-11 20:00:27,744 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 2023-02-11 20:00:27,744 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f 2023-02-11 20:00:27,744 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 2023-02-11 20:00:27,961 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 20:00:27,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } is done 2023-02-11 20:00:27,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(404): Snapshoting '{ ss=snaptb0-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 }' is still in progress! 2023-02-11 20:00:27,963 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-11 20:00:28,196 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676145622835 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } as finished. 2023-02-11 20:00:28,196 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676145622835 2023-02-11 20:00:28,197 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/snaptb0-1676145622835 2023-02-11 20:00:28,267 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/snaptb0-1676145622835 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/snaptb0-1676145622835 2023-02-11 20:00:28,303 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676145622835 of table testRestoreSnapshot-1676145622835 completed 2023-02-11 20:00:28,304 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/snaptb0-1676145622835 2023-02-11 20:00:28,304 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/snaptb0-1676145622835 2023-02-11 20:00:28,304 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676145622835 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676145622835 at 1676145627711Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676145622835 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } at 1676145627729 (+18 ms)Consolidate snapshot: snaptb0-1676145622835 at 1676145628196 (+467 ms)Loading Region manifests for snaptb0-1676145622835 at 1676145628196Writing data manifest for snaptb0-1676145622835 at 1676145628213 (+17 ms)Verifying snapshot: snaptb0-1676145622835 at 1676145628240 (+27 ms)Snapshot snaptb0-1676145622835 of table testRestoreSnapshot-1676145622835 completed at 1676145628303 (+63 ms) 2023-02-11 20:00:28,306 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:28,308 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE in 598 msec 2023-02-11 20:00:28,463 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 20:00:28,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } is done 2023-02-11 20:00:28,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 20:00:28,466 INFO [Listener at localhost/33691] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676145622835 2023-02-11 20:00:28,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676145622835 2023-02-11 20:00:28,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:28,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-11 20:00:28,471 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676145622835 2023-02-11 20:00:28,471 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145628471"}]},"ts":"1676145628471"} 2023-02-11 20:00:28,472 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=ENABLING in hbase:meta 2023-02-11 20:00:28,481 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 20:00:28,482 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN}] 2023-02-11 20:00:28,484 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN 2023-02-11 20:00:28,484 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN 2023-02-11 20:00:28,484 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN 2023-02-11 20:00:28,484 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN 2023-02-11 20:00:28,485 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN 2023-02-11 20:00:28,486 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:28,486 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:28,486 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44175,1676145619924; forceNewPlan=false, retain=true 2023-02-11 20:00:28,486 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35937,1676145619887; forceNewPlan=false, retain=true 2023-02-11 20:00:28,486 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:28,486 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42011,1676145619841; forceNewPlan=false, retain=true 2023-02-11 20:00:28,488 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:28,488 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN 2023-02-11 20:00:28,488 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44175,1676145619924; forceNewPlan=false, retain=true 2023-02-11 20:00:28,489 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:28,489 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35937,1676145619887; forceNewPlan=false, retain=true 2023-02-11 20:00:28,489 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:28,489 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42011,1676145619841; forceNewPlan=false, retain=true 2023-02-11 20:00:28,636 INFO [jenkins-hbase4:39971] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 20:00:28,637 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:28,637 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:28,637 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145628637"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145628637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145628637"}]},"ts":"1676145628637"} 2023-02-11 20:00:28,637 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:28,637 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:28,637 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:28,637 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145628637"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145628637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145628637"}]},"ts":"1676145628637"} 2023-02-11 20:00:28,637 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145628637"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145628637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145628637"}]},"ts":"1676145628637"} 2023-02-11 20:00:28,637 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145628637"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145628637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145628637"}]},"ts":"1676145628637"} 2023-02-11 20:00:28,637 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145628637"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145628637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145628637"}]},"ts":"1676145628637"} 2023-02-11 20:00:28,639 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=66, state=RUNNABLE; OpenRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:28,640 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=67, state=RUNNABLE; OpenRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:28,644 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=64, state=RUNNABLE; OpenRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:28,646 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=69, state=RUNNABLE; OpenRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:28,651 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=68, state=RUNNABLE; OpenRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:28,653 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:28,653 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145628653"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145628653"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145628653"}]},"ts":"1676145628653"} 2023-02-11 20:00:28,657 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=65, state=RUNNABLE; OpenRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:28,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-11 20:00:28,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:28,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:28,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:28,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 20:00:28,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 20:00:28,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 20:00:28,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:28,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:28,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:28,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:28,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:28,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:28,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:28,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:28,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:28,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:28,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:28,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:28,807 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:28,807 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:28,808 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:28,808 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf 2023-02-11 20:00:28,808 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf 2023-02-11 20:00:28,808 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf 2023-02-11 20:00:28,809 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf 2023-02-11 20:00:28,809 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a4676c43c8662115dfd922e9ff72451 columnFamilyName cf 2023-02-11 20:00:28,809 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8de911ca888e28adfca434ff163cbba9 columnFamilyName cf 2023-02-11 20:00:28,809 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf 2023-02-11 20:00:28,809 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf 2023-02-11 20:00:28,810 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c630b1709e678e0b00c070c7cac86d30 columnFamilyName cf 2023-02-11 20:00:28,824 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/cf1d617c62bd452786cb06efe103dd00 2023-02-11 20:00:28,824 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/d185ce9140d04e959efc0b58ace714af 2023-02-11 20:00:28,825 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(310): Store=8de911ca888e28adfca434ff163cbba9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:28,825 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(310): Store=c630b1709e678e0b00c070c7cac86d30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:28,825 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/0bcc1b49b65f42d389fcb662a909dbf1 2023-02-11 20:00:28,825 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(310): Store=1a4676c43c8662115dfd922e9ff72451/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:28,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:28,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:28,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:28,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:28,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:28,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:28,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:28,847 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 8de911ca888e28adfca434ff163cbba9; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70065611, jitterRate=0.044058963656425476}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:28,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:28,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 8de911ca888e28adfca434ff163cbba9: Running coprocessor pre-open hook at 1676145628805Writing region info on filesystem at 1676145628805Initializing all the Stores at 1676145628806 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145628806Cleaning up temporary data from old regions at 1676145628829 (+23 ms)Cleaning up detritus from prior splits at 1676145628830 (+1 ms)Running coprocessor post-open hooks at 1676145628847 (+17 ms)Region opened successfully at 1676145628847 2023-02-11 20:00:28,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:28,848 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1a4676c43c8662115dfd922e9ff72451; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73796125, jitterRate=0.09964795410633087}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:28,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1a4676c43c8662115dfd922e9ff72451: Running coprocessor pre-open hook at 1676145628805Writing region info on filesystem at 1676145628805Initializing all the Stores at 1676145628806 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145628806Cleaning up temporary data from old regions at 1676145628830 (+24 ms)Cleaning up detritus from prior splits at 1676145628832 (+2 ms)Running coprocessor post-open hooks at 1676145628848 (+16 ms)Region opened successfully at 1676145628848 2023-02-11 20:00:28,848 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9., pid=74, masterSystemTime=1676145628800 2023-02-11 20:00:28,849 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451., pid=73, masterSystemTime=1676145628800 2023-02-11 20:00:28,849 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c630b1709e678e0b00c070c7cac86d30; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67628612, jitterRate=0.007744848728179932}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:28,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c630b1709e678e0b00c070c7cac86d30: Running coprocessor pre-open hook at 1676145628805Writing region info on filesystem at 1676145628805Initializing all the Stores at 1676145628806 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145628806Cleaning up temporary data from old regions at 1676145628829 (+23 ms)Cleaning up detritus from prior splits at 1676145628830 (+1 ms)Running coprocessor post-open hooks at 1676145628850 (+20 ms)Region opened successfully at 1676145628850 2023-02-11 20:00:28,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30., pid=71, masterSystemTime=1676145628800 2023-02-11 20:00:28,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:28,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:28,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:28,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 20:00:28,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:28,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:28,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:28,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:28,854 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:28,854 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145628854"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145628854"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145628854"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145628854"}]},"ts":"1676145628854"} 2023-02-11 20:00:28,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:28,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:28,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:28,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 20:00:28,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:28,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:28,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:28,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:28,857 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:28,857 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145628857"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145628857"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145628857"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145628857"}]},"ts":"1676145628857"} 2023-02-11 20:00:28,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:28,858 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:28,858 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:28,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 20:00:28,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:28,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:28,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:28,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:28,860 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:28,860 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145628859"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145628859"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145628859"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145628859"}]},"ts":"1676145628859"} 2023-02-11 20:00:28,860 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:28,863 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=68 2023-02-11 20:00:28,864 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=68, state=SUCCESS; OpenRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841 in 208 msec 2023-02-11 20:00:28,872 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-11 20:00:28,873 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=69 2023-02-11 20:00:28,873 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=69, state=SUCCESS; OpenRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924 in 215 msec 2023-02-11 20:00:28,873 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-11 20:00:28,874 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN in 382 msec 2023-02-11 20:00:28,875 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=67 2023-02-11 20:00:28,875 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=67, state=SUCCESS; OpenRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887 in 223 msec 2023-02-11 20:00:28,876 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN in 391 msec 2023-02-11 20:00:28,876 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:28,877 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN in 393 msec 2023-02-11 20:00:28,880 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:28,881 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf 2023-02-11 20:00:28,881 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf 2023-02-11 20:00:28,881 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf 2023-02-11 20:00:28,881 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7c040ebb400ff6a8415e55ad85bbbe0e columnFamilyName cf 2023-02-11 20:00:28,881 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf 2023-02-11 20:00:28,882 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f1fc9bf98611ec0266597565431cc2a1 columnFamilyName cf 2023-02-11 20:00:28,883 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf 2023-02-11 20:00:28,883 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf 2023-02-11 20:00:28,884 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region db772e5c38666094ec2e3b8fb5e805c2 columnFamilyName cf 2023-02-11 20:00:28,896 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/e793845eb1f447169fd725e5a836da1b 2023-02-11 20:00:28,896 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/2e4307f987394952952cf7c59270b4e7 2023-02-11 20:00:28,896 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(310): Store=db772e5c38666094ec2e3b8fb5e805c2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:28,896 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(310): Store=7c040ebb400ff6a8415e55ad85bbbe0e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:28,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:28,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:28,898 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/f1723f83b8234b60b4690710a470498b 2023-02-11 20:00:28,898 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(310): Store=f1fc9bf98611ec0266597565431cc2a1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:28,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:28,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:28,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:28,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:28,903 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:28,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:28,905 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7c040ebb400ff6a8415e55ad85bbbe0e; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72134616, jitterRate=0.07488954067230225}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:28,905 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Running coprocessor pre-open hook at 1676145628856Writing region info on filesystem at 1676145628856Initializing all the Stores at 1676145628858 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145628858Cleaning up temporary data from old regions at 1676145628900 (+42 ms)Cleaning up detritus from prior splits at 1676145628901 (+1 ms)Running coprocessor post-open hooks at 1676145628905 (+4 ms)Region opened successfully at 1676145628905 2023-02-11 20:00:28,905 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:28,905 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened db772e5c38666094ec2e3b8fb5e805c2; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72643959, jitterRate=0.08247934281826019}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:28,905 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for db772e5c38666094ec2e3b8fb5e805c2: Running coprocessor pre-open hook at 1676145628852Writing region info on filesystem at 1676145628852Initializing all the Stores at 1676145628855 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145628855Cleaning up temporary data from old regions at 1676145628900 (+45 ms)Cleaning up detritus from prior splits at 1676145628901 (+1 ms)Running coprocessor post-open hooks at 1676145628905 (+4 ms)Region opened successfully at 1676145628905 2023-02-11 20:00:28,906 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e., pid=70, masterSystemTime=1676145628800 2023-02-11 20:00:28,906 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f1fc9bf98611ec0266597565431cc2a1; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71913762, jitterRate=0.07159855961799622}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:28,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f1fc9bf98611ec0266597565431cc2a1: Running coprocessor pre-open hook at 1676145628859Writing region info on filesystem at 1676145628859Initializing all the Stores at 1676145628862 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145628862Cleaning up temporary data from old regions at 1676145628902 (+40 ms)Cleaning up detritus from prior splits at 1676145628902Running coprocessor post-open hooks at 1676145628906 (+4 ms)Region opened successfully at 1676145628906 2023-02-11 20:00:28,907 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2., pid=72, masterSystemTime=1676145628800 2023-02-11 20:00:28,908 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1., pid=75, masterSystemTime=1676145628800 2023-02-11 20:00:28,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:28,909 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:28,910 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:28,910 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145628910"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145628910"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145628910"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145628910"}]},"ts":"1676145628910"} 2023-02-11 20:00:28,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:28,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:28,912 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:28,913 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145628912"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145628912"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145628912"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145628912"}]},"ts":"1676145628912"} 2023-02-11 20:00:28,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:28,913 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:28,914 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:28,914 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145628913"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145628913"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145628913"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145628913"}]},"ts":"1676145628913"} 2023-02-11 20:00:28,920 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=66 2023-02-11 20:00:28,920 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=66, state=SUCCESS; OpenRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924 in 275 msec 2023-02-11 20:00:28,921 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=64 2023-02-11 20:00:28,921 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=64, state=SUCCESS; OpenRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841 in 274 msec 2023-02-11 20:00:28,922 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=65 2023-02-11 20:00:28,922 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN in 438 msec 2023-02-11 20:00:28,922 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=65, state=SUCCESS; OpenRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887 in 264 msec 2023-02-11 20:00:28,923 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN in 439 msec 2023-02-11 20:00:28,924 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=65, resume processing ppid=63 2023-02-11 20:00:28,924 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN in 440 msec 2023-02-11 20:00:28,925 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145628925"}]},"ts":"1676145628925"} 2023-02-11 20:00:28,927 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=ENABLED in hbase:meta 2023-02-11 20:00:28,928 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676145622835' was successfully enabled. 2023-02-11 20:00:28,931 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676145622835 in 463 msec 2023-02-11 20:00:29,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-11 20:00:29,223 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676145622835, procId: 63 completed 2023-02-11 20:00:29,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:29,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:29,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:29,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:29,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:29,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:29,255 DEBUG [Listener at localhost/33691] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676145622835 2023-02-11 20:00:29,255 DEBUG [Listener at localhost/33691] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:29,256 INFO [Listener at localhost/33691] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 20:00:29,514 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 20:00:29,666 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 20:00:29,867 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 20:00:29,988 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 20:00:30,137 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 20:00:31,756 DEBUG [Listener at localhost/33691] zookeeper.ReadOnlyZKClient(139): Connect 0x490facbf to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:31,763 DEBUG [Listener at localhost/33691] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@290fa1b6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:31,765 DEBUG [hconnection-0x7b77b3f7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:31,770 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42188, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:31,775 DEBUG [hconnection-0x7b77b3f7-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:31,778 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45096, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:31,885 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 20:00:32,007 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 20:00:32,012 DEBUG [hconnection-0x7b77b3f7-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:32,016 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36944, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:32,174 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 20:00:32,264 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 20:00:32,421 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 20:00:32,909 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 20:00:33,999 INFO [Listener at localhost/33691] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676145622835 2023-02-11 20:00:34,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676145622835 2023-02-11 20:00:34,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:34,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-11 20:00:34,005 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145634005"}]},"ts":"1676145634005"} 2023-02-11 20:00:34,007 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=DISABLING in hbase:meta 2023-02-11 20:00:34,009 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676145622835 to state=DISABLING 2023-02-11 20:00:34,010 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN}] 2023-02-11 20:00:34,012 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN 2023-02-11 20:00:34,012 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN 2023-02-11 20:00:34,012 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN 2023-02-11 20:00:34,012 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN 2023-02-11 20:00:34,012 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN 2023-02-11 20:00:34,013 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:34,013 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:34,013 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:34,013 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145634013"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145634013"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145634013"}]},"ts":"1676145634013"} 2023-02-11 20:00:34,013 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:34,013 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145634013"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145634013"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145634013"}]},"ts":"1676145634013"} 2023-02-11 20:00:34,013 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:34,013 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145634013"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145634013"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145634013"}]},"ts":"1676145634013"} 2023-02-11 20:00:34,013 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145634013"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145634013"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145634013"}]},"ts":"1676145634013"} 2023-02-11 20:00:34,013 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145634013"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145634013"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145634013"}]},"ts":"1676145634013"} 2023-02-11 20:00:34,015 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=80, state=RUNNABLE; CloseRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:34,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=79, state=RUNNABLE; CloseRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:34,017 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=81, state=RUNNABLE; CloseRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:34,018 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=78, state=RUNNABLE; CloseRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:34,019 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=82, state=RUNNABLE; CloseRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:34,020 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN 2023-02-11 20:00:34,021 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:34,022 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145634021"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145634021"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145634021"}]},"ts":"1676145634021"} 2023-02-11 20:00:34,023 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=77, state=RUNNABLE; CloseRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:34,167 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:34,168 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f1fc9bf98611ec0266597565431cc2a1, disabling compactions & flushes 2023-02-11 20:00:34,168 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:34,168 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:34,168 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. after waiting 0 ms 2023-02-11 20:00:34,168 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:34,169 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing f1fc9bf98611ec0266597565431cc2a1 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-11 20:00:34,169 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:34,169 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:34,169 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing db772e5c38666094ec2e3b8fb5e805c2, disabling compactions & flushes 2023-02-11 20:00:34,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:34,171 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7c040ebb400ff6a8415e55ad85bbbe0e, disabling compactions & flushes 2023-02-11 20:00:34,171 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:34,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:34,171 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. after waiting 0 ms 2023-02-11 20:00:34,171 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:34,171 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:34,171 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. after waiting 0 ms 2023-02-11 20:00:34,171 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:34,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing db772e5c38666094ec2e3b8fb5e805c2 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-11 20:00:34,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7c040ebb400ff6a8415e55ad85bbbe0e 1/1 column families, dataSize=3.07 KB heapSize=6.86 KB 2023-02-11 20:00:34,199 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 2023-02-11 20:00:34,201 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 2023-02-11 20:00:34,203 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d 2023-02-11 20:00:34,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=3.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/.tmp/cf/233710642ffe449384153b4775e7e690 2023-02-11 20:00:34,215 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/.tmp/cf/85a30970077e4a559c5f7e7c79bd9336 2023-02-11 20:00:34,216 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/.tmp/cf/233710642ffe449384153b4775e7e690 as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/233710642ffe449384153b4775e7e690 2023-02-11 20:00:34,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/.tmp/cf/85a30970077e4a559c5f7e7c79bd9336 as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/85a30970077e4a559c5f7e7c79bd9336 2023-02-11 20:00:34,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/233710642ffe449384153b4775e7e690, entries=47, sequenceid=15, filesize=12.9 K 2023-02-11 20:00:34,222 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/85a30970077e4a559c5f7e7c79bd9336, entries=33, sequenceid=15, filesize=10.5 K 2023-02-11 20:00:34,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for db772e5c38666094ec2e3b8fb5e805c2 in 52ms, sequenceid=15, compaction requested=false 2023-02-11 20:00:34,227 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~3.07 KB/3147, heapSize ~6.84 KB/7008, currentSize=0 B/0 for 7c040ebb400ff6a8415e55ad85bbbe0e in 56ms, sequenceid=15, compaction requested=false 2023-02-11 20:00:34,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 20:00:34,236 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:34,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for db772e5c38666094ec2e3b8fb5e805c2: Waiting for close lock at 1676145634169Running coprocessor pre-close hooks at 1676145634169Disabling compacts and flushes for region at 1676145634169Disabling writes for close at 1676145634171 (+2 ms)Obtaining lock to block concurrent updates at 1676145634171Preparing flush snapshotting stores in db772e5c38666094ec2e3b8fb5e805c2 at 1676145634171Finished memstore snapshotting testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2., syncing WAL and waiting on mvcc, flushsize=dataSize=2209, getHeapSize=4992, getOffHeapSize=0, getCellsCount=33 at 1676145634172 (+1 ms)Flushing stores of testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. at 1676145634173 (+1 ms)Flushing db772e5c38666094ec2e3b8fb5e805c2/cf: creating writer at 1676145634174 (+1 ms)Flushing mob file db772e5c38666094ec2e3b8fb5e805c2/cf: appending metadata at 1676145634181 (+7 ms)Flushing mob file db772e5c38666094ec2e3b8fb5e805c2/cf: closing flushed file at 1676145634181Flushing db772e5c38666094ec2e3b8fb5e805c2/cf: appending metadata at 1676145634204 (+23 ms)Flushing db772e5c38666094ec2e3b8fb5e805c2/cf: closing flushed file at 1676145634205 (+1 ms)Flushing db772e5c38666094ec2e3b8fb5e805c2/cf: reopening flushed file at 1676145634219 (+14 ms)Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for db772e5c38666094ec2e3b8fb5e805c2 in 52ms, sequenceid=15, compaction requested=false at 1676145634223 (+4 ms)Writing region close event to WAL at 1676145634227 (+4 ms)Running coprocessor post-close hooks at 1676145634236 (+9 ms)Closed at 1676145634236 2023-02-11 20:00:34,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:34,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:34,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 8de911ca888e28adfca434ff163cbba9, disabling compactions & flushes 2023-02-11 20:00:34,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:34,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:34,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. after waiting 0 ms 2023-02-11 20:00:34,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:34,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 20:00:34,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 8de911ca888e28adfca434ff163cbba9 1/1 column families, dataSize=1.63 KB heapSize=3.77 KB 2023-02-11 20:00:34,239 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=CLOSED 2023-02-11 20:00:34,239 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145634239"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145634239"}]},"ts":"1676145634239"} 2023-02-11 20:00:34,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:34,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Waiting for close lock at 1676145634170Running coprocessor pre-close hooks at 1676145634170Disabling compacts and flushes for region at 1676145634170Disabling writes for close at 1676145634171 (+1 ms)Obtaining lock to block concurrent updates at 1676145634171Preparing flush snapshotting stores in 7c040ebb400ff6a8415e55ad85bbbe0e at 1676145634171Finished memstore snapshotting testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e., syncing WAL and waiting on mvcc, flushsize=dataSize=3147, getHeapSize=7008, getOffHeapSize=0, getCellsCount=47 at 1676145634172 (+1 ms)Flushing stores of testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. at 1676145634172Flushing 7c040ebb400ff6a8415e55ad85bbbe0e/cf: creating writer at 1676145634173 (+1 ms)Flushing mob file 7c040ebb400ff6a8415e55ad85bbbe0e/cf: appending metadata at 1676145634180 (+7 ms)Flushing mob file 7c040ebb400ff6a8415e55ad85bbbe0e/cf: closing flushed file at 1676145634180Flushing 7c040ebb400ff6a8415e55ad85bbbe0e/cf: appending metadata at 1676145634201 (+21 ms)Flushing 7c040ebb400ff6a8415e55ad85bbbe0e/cf: closing flushed file at 1676145634201Flushing 7c040ebb400ff6a8415e55ad85bbbe0e/cf: reopening flushed file at 1676145634217 (+16 ms)Finished flush of dataSize ~3.07 KB/3147, heapSize ~6.84 KB/7008, currentSize=0 B/0 for 7c040ebb400ff6a8415e55ad85bbbe0e in 56ms, sequenceid=15, compaction requested=false at 1676145634227 (+10 ms)Writing region close event to WAL at 1676145634234 (+7 ms)Running coprocessor post-close hooks at 1676145634240 (+6 ms)Closed at 1676145634240 2023-02-11 20:00:34,242 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:34,242 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:34,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1a4676c43c8662115dfd922e9ff72451, disabling compactions & flushes 2023-02-11 20:00:34,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:34,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:34,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. after waiting 0 ms 2023-02-11 20:00:34,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:34,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1a4676c43c8662115dfd922e9ff72451 1/1 column families, dataSize=21.92 KB heapSize=47.36 KB 2023-02-11 20:00:34,244 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=CLOSED 2023-02-11 20:00:34,244 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145634244"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145634244"}]},"ts":"1676145634244"} 2023-02-11 20:00:34,247 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=77 2023-02-11 20:00:34,247 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=77, state=SUCCESS; CloseRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841 in 219 msec 2023-02-11 20:00:34,249 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN in 237 msec 2023-02-11 20:00:34,249 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=79 2023-02-11 20:00:34,249 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=79, state=SUCCESS; CloseRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924 in 230 msec 2023-02-11 20:00:34,251 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN in 239 msec 2023-02-11 20:00:34,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-11 20:00:34,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e 2023-02-11 20:00:34,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de 2023-02-11 20:00:34,279 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=21.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/.tmp/cf/cda13ed20aa14162b1be58c678d7c72c 2023-02-11 20:00:34,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/.tmp/cf/e8a8517d5b3949c39858b0093dcf6481 2023-02-11 20:00:34,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/.tmp/cf/cda13ed20aa14162b1be58c678d7c72c as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/cda13ed20aa14162b1be58c678d7c72c 2023-02-11 20:00:34,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/.tmp/cf/e8a8517d5b3949c39858b0093dcf6481 as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/e8a8517d5b3949c39858b0093dcf6481 2023-02-11 20:00:34,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/e8a8517d5b3949c39858b0093dcf6481, entries=25, sequenceid=15, filesize=9.1 K 2023-02-11 20:00:34,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/cda13ed20aa14162b1be58c678d7c72c, entries=335, sequenceid=15, filesize=62.0 K 2023-02-11 20:00:34,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.92 KB/22441, heapSize ~47.34 KB/48480, currentSize=0 B/0 for 1a4676c43c8662115dfd922e9ff72451 in 44ms, sequenceid=15, compaction requested=false 2023-02-11 20:00:34,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.63 KB/1673, heapSize ~3.75 KB/3840, currentSize=0 B/0 for 8de911ca888e28adfca434ff163cbba9 in 48ms, sequenceid=15, compaction requested=false 2023-02-11 20:00:34,295 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 20:00:34,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:34,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1a4676c43c8662115dfd922e9ff72451: Waiting for close lock at 1676145634242Running coprocessor pre-close hooks at 1676145634242Disabling compacts and flushes for region at 1676145634242Disabling writes for close at 1676145634243 (+1 ms)Obtaining lock to block concurrent updates at 1676145634243Preparing flush snapshotting stores in 1a4676c43c8662115dfd922e9ff72451 at 1676145634243Finished memstore snapshotting testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451., syncing WAL and waiting on mvcc, flushsize=dataSize=22441, getHeapSize=48480, getOffHeapSize=0, getCellsCount=335 at 1676145634243Flushing stores of testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. at 1676145634244 (+1 ms)Flushing 1a4676c43c8662115dfd922e9ff72451/cf: creating writer at 1676145634245 (+1 ms)Flushing mob file 1a4676c43c8662115dfd922e9ff72451/cf: appending metadata at 1676145634253 (+8 ms)Flushing mob file 1a4676c43c8662115dfd922e9ff72451/cf: closing flushed file at 1676145634253Flushing 1a4676c43c8662115dfd922e9ff72451/cf: appending metadata at 1676145634265 (+12 ms)Flushing 1a4676c43c8662115dfd922e9ff72451/cf: closing flushed file at 1676145634265Flushing 1a4676c43c8662115dfd922e9ff72451/cf: reopening flushed file at 1676145634284 (+19 ms)Finished flush of dataSize ~21.92 KB/22441, heapSize ~47.34 KB/48480, currentSize=0 B/0 for 1a4676c43c8662115dfd922e9ff72451 in 44ms, sequenceid=15, compaction requested=false at 1676145634287 (+3 ms)Writing region close event to WAL at 1676145634291 (+4 ms)Running coprocessor post-close hooks at 1676145634295 (+4 ms)Closed at 1676145634296 (+1 ms) 2023-02-11 20:00:34,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 20:00:34,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:34,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 8de911ca888e28adfca434ff163cbba9: Waiting for close lock at 1676145634239Running coprocessor pre-close hooks at 1676145634239Disabling compacts and flushes for region at 1676145634239Disabling writes for close at 1676145634239Obtaining lock to block concurrent updates at 1676145634239Preparing flush snapshotting stores in 8de911ca888e28adfca434ff163cbba9 at 1676145634239Finished memstore snapshotting testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9., syncing WAL and waiting on mvcc, flushsize=dataSize=1673, getHeapSize=3840, getOffHeapSize=0, getCellsCount=25 at 1676145634239Flushing stores of testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. at 1676145634240 (+1 ms)Flushing 8de911ca888e28adfca434ff163cbba9/cf: creating writer at 1676145634241 (+1 ms)Flushing mob file 8de911ca888e28adfca434ff163cbba9/cf: appending metadata at 1676145634246 (+5 ms)Flushing mob file 8de911ca888e28adfca434ff163cbba9/cf: closing flushed file at 1676145634246Flushing 8de911ca888e28adfca434ff163cbba9/cf: appending metadata at 1676145634266 (+20 ms)Flushing 8de911ca888e28adfca434ff163cbba9/cf: closing flushed file at 1676145634266Flushing 8de911ca888e28adfca434ff163cbba9/cf: reopening flushed file at 1676145634284 (+18 ms)Finished flush of dataSize ~1.63 KB/1673, heapSize ~3.75 KB/3840, currentSize=0 B/0 for 8de911ca888e28adfca434ff163cbba9 in 48ms, sequenceid=15, compaction requested=false at 1676145634287 (+3 ms)Writing region close event to WAL at 1676145634292 (+5 ms)Running coprocessor post-close hooks at 1676145634296 (+4 ms)Closed at 1676145634297 (+1 ms) 2023-02-11 20:00:34,298 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:34,298 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=CLOSED 2023-02-11 20:00:34,299 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145634298"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145634298"}]},"ts":"1676145634298"} 2023-02-11 20:00:34,299 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:34,299 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=CLOSED 2023-02-11 20:00:34,299 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145634299"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145634299"}]},"ts":"1676145634299"} 2023-02-11 20:00:34,314 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=82 2023-02-11 20:00:34,314 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=82, state=SUCCESS; CloseRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924 in 281 msec 2023-02-11 20:00:34,314 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=81 2023-02-11 20:00:34,314 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=81, state=SUCCESS; CloseRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841 in 295 msec 2023-02-11 20:00:34,317 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN in 304 msec 2023-02-11 20:00:34,317 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN in 304 msec 2023-02-11 20:00:34,613 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/.tmp/cf/c116febc28fc423781baa2c67a3e1d2f 2023-02-11 20:00:34,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/.tmp/cf/c116febc28fc423781baa2c67a3e1d2f as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/c116febc28fc423781baa2c67a3e1d2f 2023-02-11 20:00:34,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/c116febc28fc423781baa2c67a3e1d2f, entries=26, sequenceid=15, filesize=9.3 K 2023-02-11 20:00:34,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for f1fc9bf98611ec0266597565431cc2a1 in 455ms, sequenceid=15, compaction requested=false 2023-02-11 20:00:34,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 20:00:34,634 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:34,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f1fc9bf98611ec0266597565431cc2a1: Waiting for close lock at 1676145634167Running coprocessor pre-close hooks at 1676145634167Disabling compacts and flushes for region at 1676145634167Disabling writes for close at 1676145634168 (+1 ms)Obtaining lock to block concurrent updates at 1676145634169 (+1 ms)Preparing flush snapshotting stores in f1fc9bf98611ec0266597565431cc2a1 at 1676145634169Finished memstore snapshotting testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1., syncing WAL and waiting on mvcc, flushsize=dataSize=1740, getHeapSize=3984, getOffHeapSize=0, getCellsCount=26 at 1676145634169Flushing stores of testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. at 1676145634170 (+1 ms)Flushing f1fc9bf98611ec0266597565431cc2a1/cf: creating writer at 1676145634171 (+1 ms)Flushing mob file f1fc9bf98611ec0266597565431cc2a1/cf: appending metadata at 1676145634177 (+6 ms)Flushing mob file f1fc9bf98611ec0266597565431cc2a1/cf: closing flushed file at 1676145634177Flushing f1fc9bf98611ec0266597565431cc2a1/cf: appending metadata at 1676145634202 (+25 ms)Flushing f1fc9bf98611ec0266597565431cc2a1/cf: closing flushed file at 1676145634202Flushing f1fc9bf98611ec0266597565431cc2a1/cf: reopening flushed file at 1676145634619 (+417 ms)Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for f1fc9bf98611ec0266597565431cc2a1 in 455ms, sequenceid=15, compaction requested=false at 1676145634624 (+5 ms)Writing region close event to WAL at 1676145634629 (+5 ms)Running coprocessor post-close hooks at 1676145634634 (+5 ms)Closed at 1676145634634 2023-02-11 20:00:34,636 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:34,636 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:34,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c630b1709e678e0b00c070c7cac86d30, disabling compactions & flushes 2023-02-11 20:00:34,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:34,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:34,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. after waiting 0 ms 2023-02-11 20:00:34,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:34,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c630b1709e678e0b00c070c7cac86d30 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-02-11 20:00:34,637 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=CLOSED 2023-02-11 20:00:34,637 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145634637"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145634637"}]},"ts":"1676145634637"} 2023-02-11 20:00:34,644 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=78 2023-02-11 20:00:34,644 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=78, state=SUCCESS; CloseRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887 in 622 msec 2023-02-11 20:00:34,646 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN in 634 msec 2023-02-11 20:00:34,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f 2023-02-11 20:00:34,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/.tmp/cf/5eb756b4ee3a4959a80c36b5b304212c 2023-02-11 20:00:34,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/.tmp/cf/5eb756b4ee3a4959a80c36b5b304212c as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/5eb756b4ee3a4959a80c36b5b304212c 2023-02-11 20:00:34,701 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/5eb756b4ee3a4959a80c36b5b304212c, entries=34, sequenceid=15, filesize=10.7 K 2023-02-11 20:00:34,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for c630b1709e678e0b00c070c7cac86d30 in 65ms, sequenceid=15, compaction requested=false 2023-02-11 20:00:34,710 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 20:00:34,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:34,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c630b1709e678e0b00c070c7cac86d30: Waiting for close lock at 1676145634636Running coprocessor pre-close hooks at 1676145634636Disabling compacts and flushes for region at 1676145634636Disabling writes for close at 1676145634637 (+1 ms)Obtaining lock to block concurrent updates at 1676145634637Preparing flush snapshotting stores in c630b1709e678e0b00c070c7cac86d30 at 1676145634637Finished memstore snapshotting testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30., syncing WAL and waiting on mvcc, flushsize=dataSize=2276, getHeapSize=5136, getOffHeapSize=0, getCellsCount=34 at 1676145634638 (+1 ms)Flushing stores of testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. at 1676145634638Flushing c630b1709e678e0b00c070c7cac86d30/cf: creating writer at 1676145634638Flushing mob file c630b1709e678e0b00c070c7cac86d30/cf: appending metadata at 1676145634644 (+6 ms)Flushing mob file c630b1709e678e0b00c070c7cac86d30/cf: closing flushed file at 1676145634644Flushing c630b1709e678e0b00c070c7cac86d30/cf: appending metadata at 1676145634669 (+25 ms)Flushing c630b1709e678e0b00c070c7cac86d30/cf: closing flushed file at 1676145634669Flushing c630b1709e678e0b00c070c7cac86d30/cf: reopening flushed file at 1676145634697 (+28 ms)Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for c630b1709e678e0b00c070c7cac86d30 in 65ms, sequenceid=15, compaction requested=false at 1676145634702 (+5 ms)Writing region close event to WAL at 1676145634707 (+5 ms)Running coprocessor post-close hooks at 1676145634711 (+4 ms)Closed at 1676145634712 (+1 ms) 2023-02-11 20:00:34,714 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:34,714 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=CLOSED 2023-02-11 20:00:34,714 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145634714"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145634714"}]},"ts":"1676145634714"} 2023-02-11 20:00:34,718 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=80 2023-02-11 20:00:34,718 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=80, state=SUCCESS; CloseRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887 in 701 msec 2023-02-11 20:00:34,720 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=80, resume processing ppid=76 2023-02-11 20:00:34,720 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN in 708 msec 2023-02-11 20:00:34,721 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145634721"}]},"ts":"1676145634721"} 2023-02-11 20:00:34,722 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=DISABLED in hbase:meta 2023-02-11 20:00:34,753 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676145622835 to state=DISABLED 2023-02-11 20:00:34,755 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x490facbf to 127.0.0.1:51732 2023-02-11 20:00:34,755 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:00:34,757 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676145622835 in 755 msec 2023-02-11 20:00:34,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-11 20:00:34,758 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676145622835, procId: 76 completed 2023-02-11 20:00:34,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb1-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:34,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676145634760 (current time:1676145634760). 2023-02-11 20:00:34,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 20:00:34,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(139): Connect 0x15916171 to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:34,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e64a3b6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:34,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:34,778 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43640, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:34,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15916171 to 127.0.0.1:51732 2023-02-11 20:00:34,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:00:34,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 20:00:34,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:34,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:34,784 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:34,785 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE; timeout=600000, timestamp=1676146234785 2023-02-11 20:00:34,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:34,786 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1676145622835 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676145622835 2023-02-11 20:00:34,786 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 20:00:34,786 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 20:00:34,809 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676145622835 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } 2023-02-11 20:00:34,817 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:34,817 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:34,826 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:34,826 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:34,827 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:34,827 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:34,827 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/85a30970077e4a559c5f7e7c79bd9336, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/e793845eb1f447169fd725e5a836da1b] hfiles 2023-02-11 20:00:34,827 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/85a30970077e4a559c5f7e7c79bd9336 2023-02-11 20:00:34,827 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/e793845eb1f447169fd725e5a836da1b 2023-02-11 20:00:34,828 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:34,828 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:34,829 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:34,829 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:34,829 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:34,829 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:34,829 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:34,830 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:34,832 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/0bcc1b49b65f42d389fcb662a909dbf1, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/cda13ed20aa14162b1be58c678d7c72c] hfiles 2023-02-11 20:00:34,832 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/0bcc1b49b65f42d389fcb662a909dbf1 2023-02-11 20:00:34,832 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/cda13ed20aa14162b1be58c678d7c72c 2023-02-11 20:00:34,835 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/233710642ffe449384153b4775e7e690, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/2e4307f987394952952cf7c59270b4e7] hfiles 2023-02-11 20:00:34,835 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/233710642ffe449384153b4775e7e690 2023-02-11 20:00:34,835 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/2e4307f987394952952cf7c59270b4e7 2023-02-11 20:00:34,835 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/d185ce9140d04e959efc0b58ace714af, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/e8a8517d5b3949c39858b0093dcf6481] hfiles 2023-02-11 20:00:34,835 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/d185ce9140d04e959efc0b58ace714af 2023-02-11 20:00:34,835 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/e8a8517d5b3949c39858b0093dcf6481 2023-02-11 20:00:34,836 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/c116febc28fc423781baa2c67a3e1d2f, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/f1723f83b8234b60b4690710a470498b] hfiles 2023-02-11 20:00:34,836 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/c116febc28fc423781baa2c67a3e1d2f 2023-02-11 20:00:34,836 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/f1723f83b8234b60b4690710a470498b 2023-02-11 20:00:34,837 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/5eb756b4ee3a4959a80c36b5b304212c, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/cf1d617c62bd452786cb06efe103dd00] hfiles 2023-02-11 20:00:34,837 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/5eb756b4ee3a4959a80c36b5b304212c 2023-02-11 20:00:34,838 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/cf1d617c62bd452786cb06efe103dd00 2023-02-11 20:00:34,840 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47] hfiles 2023-02-11 20:00:34,840 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 2023-02-11 20:00:34,840 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de 2023-02-11 20:00:34,840 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 2023-02-11 20:00:34,841 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 2023-02-11 20:00:34,841 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 2023-02-11 20:00:34,841 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 2023-02-11 20:00:34,841 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 2023-02-11 20:00:34,841 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d 2023-02-11 20:00:34,841 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f 2023-02-11 20:00:34,841 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e 2023-02-11 20:00:34,841 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f 2023-02-11 20:00:34,841 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 2023-02-11 20:00:34,895 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1676145622835 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } as finished. 2023-02-11 20:00:34,896 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1676145622835 2023-02-11 20:00:34,896 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/snaptb1-1676145622835 2023-02-11 20:00:34,933 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/snaptb1-1676145622835 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/snaptb1-1676145622835 2023-02-11 20:00:34,955 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1676145622835 of table testRestoreSnapshot-1676145622835 completed 2023-02-11 20:00:34,955 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/snaptb1-1676145622835 2023-02-11 20:00:34,955 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/snaptb1-1676145622835 2023-02-11 20:00:34,956 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1676145622835 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676145622835 at 1676145634786Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676145622835 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } at 1676145634809 (+23 ms)Consolidate snapshot: snaptb1-1676145622835 at 1676145634896 (+87 ms)Loading Region manifests for snaptb1-1676145622835 at 1676145634896Writing data manifest for snaptb1-1676145622835 at 1676145634901 (+5 ms)Verifying snapshot: snaptb1-1676145622835 at 1676145634914 (+13 ms)Snapshot snaptb1-1676145622835 of table testRestoreSnapshot-1676145622835 completed at 1676145634955 (+41 ms) 2023-02-11 20:00:34,958 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:34,960 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE in 176 msec 2023-02-11 20:00:35,037 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 20:00:35,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } is done 2023-02-11 20:00:35,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1676145622835 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 20:00:35,053 INFO [Listener at localhost/33691] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1676145622835-1676145635053 2023-02-11 20:00:35,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb0-1676145622835-1676145635053 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:35,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676145635054 (current time:1676145635054). 2023-02-11 20:00:35,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 20:00:35,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(139): Connect 0x666380c5 to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:35,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18bb1353, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:35,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:35,064 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43654, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:35,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x666380c5 to 127.0.0.1:51732 2023-02-11 20:00:35,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:00:35,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 20:00:35,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1676145622835-1676145635053 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:35,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:35,069 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:35,070 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE; timeout=600000, timestamp=1676146235070 2023-02-11 20:00:35,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1676145622835-1676145635053 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:35,070 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1676145622835-1676145635053 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676145622835 2023-02-11 20:00:35,070 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1676145622835-1676145635053 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 20:00:35,070 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 20:00:35,084 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676145622835-1676145635053 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } 2023-02-11 20:00:35,086 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:35,086 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:35,086 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:35,086 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:35,086 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:35,086 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:35,086 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:35,087 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:35,086 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:35,086 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:35,087 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:35,087 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:35,087 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:35,087 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:35,089 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/d185ce9140d04e959efc0b58ace714af, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/e8a8517d5b3949c39858b0093dcf6481] hfiles 2023-02-11 20:00:35,089 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/d185ce9140d04e959efc0b58ace714af 2023-02-11 20:00:35,089 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/e8a8517d5b3949c39858b0093dcf6481 2023-02-11 20:00:35,091 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/233710642ffe449384153b4775e7e690, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/2e4307f987394952952cf7c59270b4e7] hfiles 2023-02-11 20:00:35,091 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/c116febc28fc423781baa2c67a3e1d2f, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/f1723f83b8234b60b4690710a470498b] hfiles 2023-02-11 20:00:35,091 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/233710642ffe449384153b4775e7e690 2023-02-11 20:00:35,091 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/c116febc28fc423781baa2c67a3e1d2f 2023-02-11 20:00:35,091 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/2e4307f987394952952cf7c59270b4e7 2023-02-11 20:00:35,091 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/f1723f83b8234b60b4690710a470498b 2023-02-11 20:00:35,091 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/85a30970077e4a559c5f7e7c79bd9336, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/e793845eb1f447169fd725e5a836da1b] hfiles 2023-02-11 20:00:35,092 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/85a30970077e4a559c5f7e7c79bd9336 2023-02-11 20:00:35,092 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/e793845eb1f447169fd725e5a836da1b 2023-02-11 20:00:35,092 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/0bcc1b49b65f42d389fcb662a909dbf1, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/cda13ed20aa14162b1be58c678d7c72c] hfiles 2023-02-11 20:00:35,092 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/0bcc1b49b65f42d389fcb662a909dbf1 2023-02-11 20:00:35,092 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/cda13ed20aa14162b1be58c678d7c72c 2023-02-11 20:00:35,095 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/5eb756b4ee3a4959a80c36b5b304212c, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/cf1d617c62bd452786cb06efe103dd00] hfiles 2023-02-11 20:00:35,095 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/5eb756b4ee3a4959a80c36b5b304212c 2023-02-11 20:00:35,095 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/cf1d617c62bd452786cb06efe103dd00 2023-02-11 20:00:35,107 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47] hfiles 2023-02-11 20:00:35,107 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 2023-02-11 20:00:35,107 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de 2023-02-11 20:00:35,107 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 2023-02-11 20:00:35,107 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 2023-02-11 20:00:35,107 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 2023-02-11 20:00:35,107 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 2023-02-11 20:00:35,107 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 2023-02-11 20:00:35,107 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d 2023-02-11 20:00:35,107 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f 2023-02-11 20:00:35,107 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e 2023-02-11 20:00:35,107 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f 2023-02-11 20:00:35,107 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 2023-02-11 20:00:35,133 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1676145622835-1676145635053 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } as finished. 2023-02-11 20:00:35,133 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1676145622835-1676145635053 2023-02-11 20:00:35,134 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676145622835-1676145635053 2023-02-11 20:00:35,179 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676145622835-1676145635053 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/hbase-failsafe-snaptb0-1676145622835-1676145635053 2023-02-11 20:00:35,201 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1676145622835-1676145635053 of table testRestoreSnapshot-1676145622835 completed 2023-02-11 20:00:35,202 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676145622835-1676145635053 2023-02-11 20:00:35,202 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676145622835-1676145635053 2023-02-11 20:00:35,202 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1676145622835-1676145635053 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676145622835 at 1676145635070Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676145622835-1676145635053 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } at 1676145635084 (+14 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1676145622835-1676145635053 at 1676145635133 (+49 ms)Loading Region manifests for hbase-failsafe-snaptb0-1676145622835-1676145635053 at 1676145635133Writing data manifest for hbase-failsafe-snaptb0-1676145622835-1676145635053 at 1676145635142 (+9 ms)Verifying snapshot: hbase-failsafe-snaptb0-1676145622835-1676145635053 at 1676145635163 (+21 ms)Snapshot hbase-failsafe-snaptb0-1676145622835-1676145635053 of table testRestoreSnapshot-1676145622835 completed at 1676145635201 (+38 ms) 2023-02-11 20:00:35,205 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:35,207 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE in 138 msec 2023-02-11 20:00:35,320 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 20:00:35,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676145622835-1676145635053 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } is done 2023-02-11 20:00:35,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1676145622835-1676145635053 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 20:00:35,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676145622835 snapshot=name: "snaptb0-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145627695 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 20:00:35,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1676145622835 as table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:35,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-11 20:00:35,375 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/.tabledesc/.tableinfo.0000000002 2023-02-11 20:00:35,376 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/.tabledesc/.tableinfo.0000000001 2023-02-11 20:00:35,376 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/.tabledesc/.tableinfo.0000000002 2023-02-11 20:00:35,377 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1676145622835 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } 2023-02-11 20:00:35,381 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145627695 type: DISABLED version: 2 ttl: 0 2023-02-11 20:00:35,382 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835 2023-02-11 20:00:35,396 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:35,403 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de 2023-02-11 20:00:35,404 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e 2023-02-11 20:00:35,406 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 2023-02-11 20:00:35,408 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d 2023-02-11 20:00:35,409 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 2023-02-11 20:00:35,410 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f 2023-02-11 20:00:35,410 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:35,411 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:35,411 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:35,411 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:35,411 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:35,411 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:35,417 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/cda13ed20aa14162b1be58c678d7c72c to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/cda13ed20aa14162b1be58c678d7c72c 2023-02-11 20:00:35,418 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/233710642ffe449384153b4775e7e690 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/233710642ffe449384153b4775e7e690 2023-02-11 20:00:35,418 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/c116febc28fc423781baa2c67a3e1d2f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/c116febc28fc423781baa2c67a3e1d2f 2023-02-11 20:00:35,418 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/5eb756b4ee3a4959a80c36b5b304212c to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/5eb756b4ee3a4959a80c36b5b304212c 2023-02-11 20:00:35,419 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/e8a8517d5b3949c39858b0093dcf6481 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/e8a8517d5b3949c39858b0093dcf6481 2023-02-11 20:00:35,419 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/85a30970077e4a559c5f7e7c79bd9336 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/85a30970077e4a559c5f7e7c79bd9336 2023-02-11 20:00:35,419 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145627695 type: DISABLED version: 2 ttl: 0 2023-02-11 20:00:35,421 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145635421"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:35,421 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145635421"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:35,421 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145635421"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:35,421 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145635421"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:35,421 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145635421"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:35,421 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145635421"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:35,424 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 20:00:35,424 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-11 20:00:35,425 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145635422"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145635422"}]},"ts":"1676145635422"} 2023-02-11 20:00:35,425 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145635422"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145635422"}]},"ts":"1676145635422"} 2023-02-11 20:00:35,425 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145635422"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145635422"}]},"ts":"1676145635422"} 2023-02-11 20:00:35,425 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145635422"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145635422"}]},"ts":"1676145635422"} 2023-02-11 20:00:35,425 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145635422"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145635422"}]},"ts":"1676145635422"} 2023-02-11 20:00:35,425 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145635422"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145635422"}]},"ts":"1676145635422"} 2023-02-11 20:00:35,427 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 20:00:35,427 INFO [PEWorker-4] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-11 20:00:35,427 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-11 20:00:35,427 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1676145622835 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676145622835 completed! 2023-02-11 20:00:35,430 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676145622835 snapshot=name: "snaptb0-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145627695 type: DISABLED version: 2 ttl: 0 ) in 84 msec 2023-02-11 20:00:35,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-11 20:00:35,612 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676145622835, procId: 91 completed 2023-02-11 20:00:35,612 INFO [Listener at localhost/33691] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1676145622835-1676145635053 2023-02-11 20:00:35,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb0-1676145622835-1676145635053" 2023-02-11 20:00:35,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1676145622835-1676145635053 2023-02-11 20:00:35,621 INFO [Listener at localhost/33691] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676145622835 2023-02-11 20:00:35,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676145622835 2023-02-11 20:00:35,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:35,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-11 20:00:35,625 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676145622835 2023-02-11 20:00:35,626 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145635625"}]},"ts":"1676145635625"} 2023-02-11 20:00:35,627 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=ENABLING in hbase:meta 2023-02-11 20:00:35,629 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 20:00:35,631 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN}] 2023-02-11 20:00:35,633 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN 2023-02-11 20:00:35,633 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN 2023-02-11 20:00:35,633 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN 2023-02-11 20:00:35,633 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN 2023-02-11 20:00:35,634 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:35,634 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:35,634 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:35,634 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:35,635 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN 2023-02-11 20:00:35,635 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN 2023-02-11 20:00:35,635 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:35,635 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:35,784 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 20:00:35,785 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 20:00:35,785 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 20:00:35,785 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 20:00:35,785 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 20:00:35,788 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:35,788 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:35,788 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:35,788 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145635788"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145635788"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145635788"}]},"ts":"1676145635788"} 2023-02-11 20:00:35,788 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:35,788 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:35,788 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145635788"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145635788"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145635788"}]},"ts":"1676145635788"} 2023-02-11 20:00:35,788 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145635787"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145635787"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145635787"}]},"ts":"1676145635787"} 2023-02-11 20:00:35,788 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145635787"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145635787"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145635787"}]},"ts":"1676145635787"} 2023-02-11 20:00:35,788 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145635788"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145635788"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145635788"}]},"ts":"1676145635788"} 2023-02-11 20:00:35,790 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=97, state=RUNNABLE; OpenRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:35,790 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=94, state=RUNNABLE; OpenRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:35,792 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=98, state=RUNNABLE; OpenRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:35,792 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=95, state=RUNNABLE; OpenRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:35,793 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=96, state=RUNNABLE; OpenRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:35,796 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:35,796 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145635796"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145635796"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145635796"}]},"ts":"1676145635796"} 2023-02-11 20:00:35,798 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=93, state=RUNNABLE; OpenRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:35,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-11 20:00:35,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:35,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:35,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 20:00:35,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 20:00:35,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:35,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 20:00:35,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:35,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:35,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:35,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:35,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:35,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:35,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:35,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:35,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:35,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:35,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:35,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:35,952 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:35,952 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:35,952 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:35,953 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf 2023-02-11 20:00:35,953 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf 2023-02-11 20:00:35,953 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf 2023-02-11 20:00:35,953 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf 2023-02-11 20:00:35,953 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf 2023-02-11 20:00:35,953 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf 2023-02-11 20:00:35,954 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region db772e5c38666094ec2e3b8fb5e805c2 columnFamilyName cf 2023-02-11 20:00:35,954 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a4676c43c8662115dfd922e9ff72451 columnFamilyName cf 2023-02-11 20:00:35,954 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8de911ca888e28adfca434ff163cbba9 columnFamilyName cf 2023-02-11 20:00:35,958 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/d185ce9140d04e959efc0b58ace714af 2023-02-11 20:00:35,959 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/e793845eb1f447169fd725e5a836da1b 2023-02-11 20:00:35,959 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(310): Store=8de911ca888e28adfca434ff163cbba9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:35,959 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(310): Store=db772e5c38666094ec2e3b8fb5e805c2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:35,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:35,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:35,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:35,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:35,962 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/0bcc1b49b65f42d389fcb662a909dbf1 2023-02-11 20:00:35,962 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(310): Store=1a4676c43c8662115dfd922e9ff72451/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:35,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:35,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:35,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:35,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:35,964 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 8de911ca888e28adfca434ff163cbba9; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70003380, jitterRate=0.04313164949417114}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:35,964 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened db772e5c38666094ec2e3b8fb5e805c2; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64144964, jitterRate=-0.04416555166244507}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:35,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 8de911ca888e28adfca434ff163cbba9: Running coprocessor pre-open hook at 1676145635948Writing region info on filesystem at 1676145635948Initializing all the Stores at 1676145635949 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145635949Cleaning up temporary data from old regions at 1676145635961 (+12 ms)Cleaning up detritus from prior splits at 1676145635962 (+1 ms)Running coprocessor post-open hooks at 1676145635964 (+2 ms)Region opened successfully at 1676145635964 2023-02-11 20:00:35,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for db772e5c38666094ec2e3b8fb5e805c2: Running coprocessor pre-open hook at 1676145635948Writing region info on filesystem at 1676145635948Initializing all the Stores at 1676145635949 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145635949Cleaning up temporary data from old regions at 1676145635961 (+12 ms)Cleaning up detritus from prior splits at 1676145635962 (+1 ms)Running coprocessor post-open hooks at 1676145635964 (+2 ms)Region opened successfully at 1676145635964 2023-02-11 20:00:35,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9., pid=99, masterSystemTime=1676145635944 2023-02-11 20:00:35,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2., pid=104, masterSystemTime=1676145635945 2023-02-11 20:00:35,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:35,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:35,967 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:35,967 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:35,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 20:00:35,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:35,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:35,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:35,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:35,968 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:35,968 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145635968"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145635968"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145635968"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145635968"}]},"ts":"1676145635968"} 2023-02-11 20:00:35,968 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1a4676c43c8662115dfd922e9ff72451; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66146911, jitterRate=-0.014334216713905334}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:35,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1a4676c43c8662115dfd922e9ff72451: Running coprocessor pre-open hook at 1676145635948Writing region info on filesystem at 1676145635948Initializing all the Stores at 1676145635949 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145635949Cleaning up temporary data from old regions at 1676145635964 (+15 ms)Cleaning up detritus from prior splits at 1676145635964Running coprocessor post-open hooks at 1676145635968 (+4 ms)Region opened successfully at 1676145635968 2023-02-11 20:00:35,969 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:35,969 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145635969"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145635969"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145635969"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145635969"}]},"ts":"1676145635969"} 2023-02-11 20:00:35,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451., pid=101, masterSystemTime=1676145635944 2023-02-11 20:00:35,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:35,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:35,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:35,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 20:00:35,969 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:35,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:35,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:35,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:35,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:35,970 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf 2023-02-11 20:00:35,970 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf 2023-02-11 20:00:35,971 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c630b1709e678e0b00c070c7cac86d30 columnFamilyName cf 2023-02-11 20:00:35,972 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:35,972 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:35,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:35,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:35,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 20:00:35,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:35,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:35,973 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf 2023-02-11 20:00:35,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:35,973 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf 2023-02-11 20:00:35,973 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:35,974 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145635973"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145635973"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145635973"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145635973"}]},"ts":"1676145635973"} 2023-02-11 20:00:35,974 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f1fc9bf98611ec0266597565431cc2a1 columnFamilyName cf 2023-02-11 20:00:35,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:35,973 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=93 2023-02-11 20:00:35,974 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=93, state=SUCCESS; OpenRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,35937,1676145619887 in 172 msec 2023-02-11 20:00:35,975 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=97 2023-02-11 20:00:35,975 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=97, state=SUCCESS; OpenRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841 in 181 msec 2023-02-11 20:00:35,976 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN in 345 msec 2023-02-11 20:00:35,976 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN in 344 msec 2023-02-11 20:00:35,977 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=98 2023-02-11 20:00:35,977 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=98, state=SUCCESS; OpenRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924 in 184 msec 2023-02-11 20:00:35,978 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN in 346 msec 2023-02-11 20:00:35,980 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:35,981 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf 2023-02-11 20:00:35,981 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf 2023-02-11 20:00:35,981 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7c040ebb400ff6a8415e55ad85bbbe0e columnFamilyName cf 2023-02-11 20:00:35,985 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/f1723f83b8234b60b4690710a470498b 2023-02-11 20:00:35,985 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(310): Store=f1fc9bf98611ec0266597565431cc2a1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:35,985 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/cf1d617c62bd452786cb06efe103dd00 2023-02-11 20:00:35,986 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(310): Store=c630b1709e678e0b00c070c7cac86d30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:35,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:35,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:35,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:35,987 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/2e4307f987394952952cf7c59270b4e7 2023-02-11 20:00:35,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:35,987 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(310): Store=7c040ebb400ff6a8415e55ad85bbbe0e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:35,988 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:35,990 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:35,990 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:35,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:35,991 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f1fc9bf98611ec0266597565431cc2a1; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61459848, jitterRate=-0.08417689800262451}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:35,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f1fc9bf98611ec0266597565431cc2a1: Running coprocessor pre-open hook at 1676145635970Writing region info on filesystem at 1676145635970Initializing all the Stores at 1676145635971 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145635971Cleaning up temporary data from old regions at 1676145635987 (+16 ms)Cleaning up detritus from prior splits at 1676145635988 (+1 ms)Running coprocessor post-open hooks at 1676145635991 (+3 ms)Region opened successfully at 1676145635991 2023-02-11 20:00:35,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c630b1709e678e0b00c070c7cac86d30; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73237626, jitterRate=0.09132567048072815}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:35,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1., pid=100, masterSystemTime=1676145635944 2023-02-11 20:00:35,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c630b1709e678e0b00c070c7cac86d30: Running coprocessor pre-open hook at 1676145635968Writing region info on filesystem at 1676145635968Initializing all the Stores at 1676145635969 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145635969Cleaning up temporary data from old regions at 1676145635988 (+19 ms)Cleaning up detritus from prior splits at 1676145635988Running coprocessor post-open hooks at 1676145635992 (+4 ms)Region opened successfully at 1676145635992 2023-02-11 20:00:35,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:35,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30., pid=103, masterSystemTime=1676145635945 2023-02-11 20:00:35,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:35,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:35,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7c040ebb400ff6a8415e55ad85bbbe0e; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72189309, jitterRate=0.07570452988147736}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:35,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Running coprocessor pre-open hook at 1676145635974Writing region info on filesystem at 1676145635974Initializing all the Stores at 1676145635975 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145635975Cleaning up temporary data from old regions at 1676145635990 (+15 ms)Cleaning up detritus from prior splits at 1676145635991 (+1 ms)Running coprocessor post-open hooks at 1676145635994 (+3 ms)Region opened successfully at 1676145635994 2023-02-11 20:00:35,994 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:35,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:35,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:35,994 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145635994"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145635994"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145635994"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145635994"}]},"ts":"1676145635994"} 2023-02-11 20:00:35,995 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e., pid=102, masterSystemTime=1676145635944 2023-02-11 20:00:35,995 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:35,995 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145635995"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145635995"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145635995"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145635995"}]},"ts":"1676145635995"} 2023-02-11 20:00:35,996 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:35,996 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:35,996 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:35,996 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145635996"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145635996"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145635996"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145635996"}]},"ts":"1676145635996"} 2023-02-11 20:00:35,998 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=94 2023-02-11 20:00:35,998 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=94, state=SUCCESS; OpenRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,42011,1676145619841 in 206 msec 2023-02-11 20:00:35,999 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=96 2023-02-11 20:00:35,999 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=96, state=SUCCESS; OpenRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887 in 204 msec 2023-02-11 20:00:35,999 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN in 369 msec 2023-02-11 20:00:36,000 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=95 2023-02-11 20:00:36,000 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=95, state=SUCCESS; OpenRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924 in 206 msec 2023-02-11 20:00:36,000 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN in 368 msec 2023-02-11 20:00:36,002 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=95, resume processing ppid=92 2023-02-11 20:00:36,002 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN in 369 msec 2023-02-11 20:00:36,002 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145636002"}]},"ts":"1676145636002"} 2023-02-11 20:00:36,003 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=ENABLED in hbase:meta 2023-02-11 20:00:36,006 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676145622835' was successfully enabled. 2023-02-11 20:00:36,008 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676145622835 in 385 msec 2023-02-11 20:00:36,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-11 20:00:36,377 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676145622835, procId: 92 completed 2023-02-11 20:00:36,378 DEBUG [Listener at localhost/33691] zookeeper.ReadOnlyZKClient(139): Connect 0x251148a9 to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:36,401 DEBUG [Listener at localhost/33691] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c1707f4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:36,403 DEBUG [hconnection-0x29e7c4cf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:36,405 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43666, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:36,492 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 20:00:36,495 DEBUG [hconnection-0x29e7c4cf-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:36,499 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53814, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:36,588 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 20:00:36,590 DEBUG [hconnection-0x29e7c4cf-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:36,595 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58562, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:36,701 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 20:00:36,755 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 20:00:36,845 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 20:00:38,030 INFO [Listener at localhost/33691] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676145622835 2023-02-11 20:00:38,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676145622835 2023-02-11 20:00:38,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:38,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-11 20:00:38,035 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145638035"}]},"ts":"1676145638035"} 2023-02-11 20:00:38,036 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=DISABLING in hbase:meta 2023-02-11 20:00:38,038 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676145622835 to state=DISABLING 2023-02-11 20:00:38,039 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN}] 2023-02-11 20:00:38,041 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN 2023-02-11 20:00:38,041 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN 2023-02-11 20:00:38,042 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN 2023-02-11 20:00:38,042 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN 2023-02-11 20:00:38,042 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN 2023-02-11 20:00:38,042 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:38,043 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145638042"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145638042"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145638042"}]},"ts":"1676145638042"} 2023-02-11 20:00:38,043 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:38,043 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:38,043 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145638043"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145638043"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145638043"}]},"ts":"1676145638043"} 2023-02-11 20:00:38,043 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:38,043 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145638043"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145638043"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145638043"}]},"ts":"1676145638043"} 2023-02-11 20:00:38,043 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145638043"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145638043"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145638043"}]},"ts":"1676145638043"} 2023-02-11 20:00:38,043 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:38,043 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145638043"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145638043"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145638043"}]},"ts":"1676145638043"} 2023-02-11 20:00:38,044 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=108, state=RUNNABLE; CloseRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:38,047 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=109, state=RUNNABLE; CloseRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:38,048 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=107, state=RUNNABLE; CloseRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:38,049 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=106, state=RUNNABLE; CloseRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:38,050 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=111, state=RUNNABLE; CloseRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:38,051 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN 2023-02-11 20:00:38,052 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:38,052 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145638052"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145638052"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145638052"}]},"ts":"1676145638052"} 2023-02-11 20:00:38,054 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=110, state=RUNNABLE; CloseRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:38,199 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:38,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1a4676c43c8662115dfd922e9ff72451, disabling compactions & flushes 2023-02-11 20:00:38,201 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:38,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:38,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. after waiting 0 ms 2023-02-11 20:00:38,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:38,201 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:38,201 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:38,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f1fc9bf98611ec0266597565431cc2a1, disabling compactions & flushes 2023-02-11 20:00:38,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing db772e5c38666094ec2e3b8fb5e805c2, disabling compactions & flushes 2023-02-11 20:00:38,203 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:38,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:38,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. after waiting 0 ms 2023-02-11 20:00:38,205 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:38,203 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:38,205 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:38,205 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. after waiting 0 ms 2023-02-11 20:00:38,205 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:38,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 20:00:38,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 20:00:38,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 20:00:38,217 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:38,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f1fc9bf98611ec0266597565431cc2a1: Waiting for close lock at 1676145638201Running coprocessor pre-close hooks at 1676145638201Disabling compacts and flushes for region at 1676145638201Disabling writes for close at 1676145638205 (+4 ms)Writing region close event to WAL at 1676145638207 (+2 ms)Running coprocessor post-close hooks at 1676145638217 (+10 ms)Closed at 1676145638217 2023-02-11 20:00:38,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:38,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1a4676c43c8662115dfd922e9ff72451: Waiting for close lock at 1676145638200Running coprocessor pre-close hooks at 1676145638200Disabling compacts and flushes for region at 1676145638200Disabling writes for close at 1676145638201 (+1 ms)Writing region close event to WAL at 1676145638205 (+4 ms)Running coprocessor post-close hooks at 1676145638218 (+13 ms)Closed at 1676145638218 2023-02-11 20:00:38,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:38,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for db772e5c38666094ec2e3b8fb5e805c2: Waiting for close lock at 1676145638203Running coprocessor pre-close hooks at 1676145638203Disabling compacts and flushes for region at 1676145638203Disabling writes for close at 1676145638205 (+2 ms)Writing region close event to WAL at 1676145638214 (+9 ms)Running coprocessor post-close hooks at 1676145638218 (+4 ms)Closed at 1676145638218 2023-02-11 20:00:38,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:38,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:38,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 8de911ca888e28adfca434ff163cbba9, disabling compactions & flushes 2023-02-11 20:00:38,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:38,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:38,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. after waiting 0 ms 2023-02-11 20:00:38,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:38,219 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=CLOSED 2023-02-11 20:00:38,220 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145638219"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145638219"}]},"ts":"1676145638219"} 2023-02-11 20:00:38,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:38,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:38,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7c040ebb400ff6a8415e55ad85bbbe0e, disabling compactions & flushes 2023-02-11 20:00:38,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:38,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:38,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. after waiting 0 ms 2023-02-11 20:00:38,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:38,220 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=CLOSED 2023-02-11 20:00:38,220 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145638220"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145638220"}]},"ts":"1676145638220"} 2023-02-11 20:00:38,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:38,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:38,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c630b1709e678e0b00c070c7cac86d30, disabling compactions & flushes 2023-02-11 20:00:38,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:38,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:38,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. after waiting 0 ms 2023-02-11 20:00:38,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:38,222 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=CLOSED 2023-02-11 20:00:38,222 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145638222"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145638222"}]},"ts":"1676145638222"} 2023-02-11 20:00:38,225 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 20:00:38,225 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 20:00:38,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:38,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 20:00:38,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 8de911ca888e28adfca434ff163cbba9: Waiting for close lock at 1676145638219Running coprocessor pre-close hooks at 1676145638219Disabling compacts and flushes for region at 1676145638219Disabling writes for close at 1676145638219Writing region close event to WAL at 1676145638220 (+1 ms)Running coprocessor post-close hooks at 1676145638226 (+6 ms)Closed at 1676145638226 2023-02-11 20:00:38,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:38,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Waiting for close lock at 1676145638220Running coprocessor pre-close hooks at 1676145638220Disabling compacts and flushes for region at 1676145638220Disabling writes for close at 1676145638220Writing region close event to WAL at 1676145638221 (+1 ms)Running coprocessor post-close hooks at 1676145638226 (+5 ms)Closed at 1676145638226 2023-02-11 20:00:38,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:38,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c630b1709e678e0b00c070c7cac86d30: Waiting for close lock at 1676145638221Running coprocessor pre-close hooks at 1676145638221Disabling compacts and flushes for region at 1676145638221Disabling writes for close at 1676145638221Writing region close event to WAL at 1676145638222 (+1 ms)Running coprocessor post-close hooks at 1676145638226 (+4 ms)Closed at 1676145638226 2023-02-11 20:00:38,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:38,233 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=CLOSED 2023-02-11 20:00:38,233 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145638233"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145638233"}]},"ts":"1676145638233"} 2023-02-11 20:00:38,233 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:38,234 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=107 2023-02-11 20:00:38,234 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=107, state=SUCCESS; CloseRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,42011,1676145619841 in 174 msec 2023-02-11 20:00:38,234 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=111 2023-02-11 20:00:38,234 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=CLOSED 2023-02-11 20:00:38,234 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=106 2023-02-11 20:00:38,234 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=111, state=SUCCESS; CloseRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,44175,1676145619924 in 173 msec 2023-02-11 20:00:38,235 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=106, state=SUCCESS; CloseRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,35937,1676145619887 in 174 msec 2023-02-11 20:00:38,235 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145638234"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145638234"}]},"ts":"1676145638234"} 2023-02-11 20:00:38,235 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:38,236 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=CLOSED 2023-02-11 20:00:38,236 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145638235"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145638235"}]},"ts":"1676145638235"} 2023-02-11 20:00:38,236 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN in 196 msec 2023-02-11 20:00:38,237 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN in 196 msec 2023-02-11 20:00:38,237 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=110 2023-02-11 20:00:38,238 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=110, state=SUCCESS; CloseRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841 in 181 msec 2023-02-11 20:00:38,238 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN in 198 msec 2023-02-11 20:00:38,239 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN in 199 msec 2023-02-11 20:00:38,240 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=108 2023-02-11 20:00:38,240 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=108, state=SUCCESS; CloseRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924 in 194 msec 2023-02-11 20:00:38,240 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=109 2023-02-11 20:00:38,240 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=109, state=SUCCESS; CloseRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887 in 193 msec 2023-02-11 20:00:38,241 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN in 201 msec 2023-02-11 20:00:38,242 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=109, resume processing ppid=105 2023-02-11 20:00:38,242 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN in 201 msec 2023-02-11 20:00:38,243 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145638242"}]},"ts":"1676145638242"} 2023-02-11 20:00:38,244 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=DISABLED in hbase:meta 2023-02-11 20:00:38,245 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676145622835 to state=DISABLED 2023-02-11 20:00:38,247 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676145622835 in 215 msec 2023-02-11 20:00:38,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-11 20:00:38,286 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676145622835, procId: 105 completed 2023-02-11 20:00:38,297 INFO [Listener at localhost/33691] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676145622835-1676145638297 2023-02-11 20:00:38,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1676145622835-1676145638297 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:38,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676145638298 (current time:1676145638298). 2023-02-11 20:00:38,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 20:00:38,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(139): Connect 0x6fe8d00a to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:38,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7013beba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:38,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:38,310 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43670, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:38,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fe8d00a to 127.0.0.1:51732 2023-02-11 20:00:38,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:00:38,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 20:00:38,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1676145622835-1676145638297 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:38,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:38,315 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:38,316 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE; timeout=600000, timestamp=1676146238316 2023-02-11 20:00:38,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1676145622835-1676145638297 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:38,316 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676145622835-1676145638297 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676145622835 2023-02-11 20:00:38,317 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1676145622835-1676145638297 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 20:00:38,317 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 20:00:38,333 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676145622835-1676145638297 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } 2023-02-11 20:00:38,341 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:38,341 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:38,341 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:38,341 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:38,342 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:38,342 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:38,342 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:38,342 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:38,343 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:38,343 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:38,343 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:38,343 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:38,345 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/e793845eb1f447169fd725e5a836da1b] hfiles 2023-02-11 20:00:38,345 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:38,345 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:38,345 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/e793845eb1f447169fd725e5a836da1b 2023-02-11 20:00:38,346 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/0bcc1b49b65f42d389fcb662a909dbf1] hfiles 2023-02-11 20:00:38,348 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/0bcc1b49b65f42d389fcb662a909dbf1 2023-02-11 20:00:38,346 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/2e4307f987394952952cf7c59270b4e7] hfiles 2023-02-11 20:00:38,348 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/2e4307f987394952952cf7c59270b4e7 2023-02-11 20:00:38,349 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/f1723f83b8234b60b4690710a470498b] hfiles 2023-02-11 20:00:38,349 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/f1723f83b8234b60b4690710a470498b 2023-02-11 20:00:38,349 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/cf1d617c62bd452786cb06efe103dd00] hfiles 2023-02-11 20:00:38,350 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/d185ce9140d04e959efc0b58ace714af] hfiles 2023-02-11 20:00:38,350 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/d185ce9140d04e959efc0b58ace714af 2023-02-11 20:00:38,350 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/cf1d617c62bd452786cb06efe103dd00 2023-02-11 20:00:38,367 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47] hfiles 2023-02-11 20:00:38,367 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 2023-02-11 20:00:38,367 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 2023-02-11 20:00:38,367 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 2023-02-11 20:00:38,367 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 2023-02-11 20:00:38,367 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f 2023-02-11 20:00:38,367 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 2023-02-11 20:00:38,430 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1676145622835-1676145638297 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } as finished. 2023-02-11 20:00:38,430 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1676145622835-1676145638297 2023-02-11 20:00:38,431 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676145622835-1676145638297 2023-02-11 20:00:38,469 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676145622835-1676145638297 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/hbase-failsafe-emptySnaptb-1676145622835-1676145638297 2023-02-11 20:00:38,502 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1676145622835-1676145638297 of table testRestoreSnapshot-1676145622835 completed 2023-02-11 20:00:38,502 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676145622835-1676145638297 2023-02-11 20:00:38,502 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676145622835-1676145638297 2023-02-11 20:00:38,502 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676145622835-1676145638297 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676145622835 at 1676145638316Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676145622835-1676145638297 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } at 1676145638333 (+17 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1676145622835-1676145638297 at 1676145638430 (+97 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1676145622835-1676145638297 at 1676145638430Writing data manifest for hbase-failsafe-emptySnaptb-1676145622835-1676145638297 at 1676145638439 (+9 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1676145622835-1676145638297 at 1676145638459 (+20 ms)Snapshot hbase-failsafe-emptySnaptb-1676145622835-1676145638297 of table testRestoreSnapshot-1676145622835 completed at 1676145638502 (+43 ms) 2023-02-11 20:00:38,504 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:38,506 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE in 190 msec 2023-02-11 20:00:38,567 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 20:00:38,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676145622835-1676145638297 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } is done 2023-02-11 20:00:38,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1676145622835-1676145638297 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 20:00:38,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676145622835 snapshot=name: "emptySnaptb-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145625259 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 20:00:38,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1676145622835 as table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:38,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-11 20:00:38,596 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/.tabledesc/.tableinfo.0000000003 2023-02-11 20:00:38,597 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/.tabledesc/.tableinfo.0000000002 2023-02-11 20:00:38,597 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/.tabledesc/.tableinfo.0000000003 2023-02-11 20:00:38,598 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1676145622835 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } 2023-02-11 20:00:38,600 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145625259 type: DISABLED version: 2 ttl: 0 2023-02-11 20:00:38,600 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835 2023-02-11 20:00:38,608 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:38,612 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 2023-02-11 20:00:38,613 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 2023-02-11 20:00:38,614 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 2023-02-11 20:00:38,616 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 2023-02-11 20:00:38,617 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f 2023-02-11 20:00:38,618 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 2023-02-11 20:00:38,619 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:38,619 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:38,619 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:38,619 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:38,619 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:38,619 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:38,632 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/d185ce9140d04e959efc0b58ace714af to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/d185ce9140d04e959efc0b58ace714af 2023-02-11 20:00:38,633 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/0bcc1b49b65f42d389fcb662a909dbf1 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/0bcc1b49b65f42d389fcb662a909dbf1 2023-02-11 20:00:38,637 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/cf1d617c62bd452786cb06efe103dd00 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/cf1d617c62bd452786cb06efe103dd00 2023-02-11 20:00:38,639 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/2e4307f987394952952cf7c59270b4e7 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/2e4307f987394952952cf7c59270b4e7 2023-02-11 20:00:38,639 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/f1723f83b8234b60b4690710a470498b to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/f1723f83b8234b60b4690710a470498b 2023-02-11 20:00:38,640 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/e793845eb1f447169fd725e5a836da1b to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/e793845eb1f447169fd725e5a836da1b 2023-02-11 20:00:38,644 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145625259 type: DISABLED version: 2 ttl: 0 2023-02-11 20:00:38,645 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145638645"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:38,645 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145638645"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:38,645 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145638645"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:38,646 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145638645"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:38,646 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145638645"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:38,646 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145638645"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:38,648 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 20:00:38,648 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-11 20:00:38,648 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145638646"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145638646"}]},"ts":"1676145638646"} 2023-02-11 20:00:38,649 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145638646"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145638646"}]},"ts":"1676145638646"} 2023-02-11 20:00:38,649 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145638646"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145638646"}]},"ts":"1676145638646"} 2023-02-11 20:00:38,649 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145638646"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145638646"}]},"ts":"1676145638646"} 2023-02-11 20:00:38,649 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145638646"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145638646"}]},"ts":"1676145638646"} 2023-02-11 20:00:38,649 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145638646"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145638646"}]},"ts":"1676145638646"} 2023-02-11 20:00:38,651 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 20:00:38,651 INFO [PEWorker-3] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-11 20:00:38,651 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-11 20:00:38,651 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1676145622835 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676145622835 completed! 2023-02-11 20:00:38,654 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676145622835 snapshot=name: "emptySnaptb-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145625259 type: DISABLED version: 2 ttl: 0 ) in 77 msec 2023-02-11 20:00:38,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-11 20:00:38,830 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676145622835, procId: 119 completed 2023-02-11 20:00:38,830 INFO [Listener at localhost/33691] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676145622835-1676145638297 2023-02-11 20:00:38,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-emptySnaptb-1676145622835-1676145638297" 2023-02-11 20:00:38,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1676145622835-1676145638297 2023-02-11 20:00:38,834 INFO [Listener at localhost/33691] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676145622835 2023-02-11 20:00:38,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676145622835 2023-02-11 20:00:38,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:38,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-11 20:00:38,838 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676145622835 2023-02-11 20:00:38,838 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145638838"}]},"ts":"1676145638838"} 2023-02-11 20:00:38,839 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=ENABLING in hbase:meta 2023-02-11 20:00:38,841 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 20:00:38,842 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN}] 2023-02-11 20:00:38,843 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN 2023-02-11 20:00:38,844 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN 2023-02-11 20:00:38,844 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN 2023-02-11 20:00:38,844 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN 2023-02-11 20:00:38,844 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN 2023-02-11 20:00:38,844 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:38,844 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:38,845 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:38,848 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:38,848 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:38,848 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN 2023-02-11 20:00:38,849 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:38,995 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 20:00:38,995 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 20:00:38,995 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 20:00:38,995 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 20:00:38,995 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 20:00:39,001 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:39,001 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:39,001 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639001"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145639001"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145639001"}]},"ts":"1676145639001"} 2023-02-11 20:00:39,001 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:39,002 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145639001"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145639001"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145639001"}]},"ts":"1676145639001"} 2023-02-11 20:00:39,001 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:39,001 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:39,002 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639001"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145639001"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145639001"}]},"ts":"1676145639001"} 2023-02-11 20:00:39,002 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639001"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145639001"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145639001"}]},"ts":"1676145639001"} 2023-02-11 20:00:39,001 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145639001"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145639001"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145639001"}]},"ts":"1676145639001"} 2023-02-11 20:00:39,003 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=125, state=RUNNABLE; OpenRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:39,004 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=126, state=RUNNABLE; OpenRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:39,005 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=123, state=RUNNABLE; OpenRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:39,006 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=124, state=RUNNABLE; OpenRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:39,008 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=121, state=RUNNABLE; OpenRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:39,008 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:39,010 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639008"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145639008"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145639008"}]},"ts":"1676145639008"} 2023-02-11 20:00:39,011 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=122, state=RUNNABLE; OpenRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:39,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-11 20:00:39,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:39,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:39,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 20:00:39,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:39,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:39,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:39,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:39,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 20:00:39,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:39,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:39,160 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:39,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:39,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:39,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 20:00:39,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:39,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:39,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:39,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:39,161 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:39,162 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:39,162 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:39,163 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf 2023-02-11 20:00:39,163 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf 2023-02-11 20:00:39,163 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf 2023-02-11 20:00:39,163 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf 2023-02-11 20:00:39,163 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf 2023-02-11 20:00:39,163 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf 2023-02-11 20:00:39,163 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f1fc9bf98611ec0266597565431cc2a1 columnFamilyName cf 2023-02-11 20:00:39,164 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a4676c43c8662115dfd922e9ff72451 columnFamilyName cf 2023-02-11 20:00:39,164 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c630b1709e678e0b00c070c7cac86d30 columnFamilyName cf 2023-02-11 20:00:39,165 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(310): Store=c630b1709e678e0b00c070c7cac86d30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:39,165 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(310): Store=1a4676c43c8662115dfd922e9ff72451/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:39,165 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(310): Store=f1fc9bf98611ec0266597565431cc2a1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:39,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:39,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:39,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:39,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:39,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:39,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:39,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:39,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:39,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:39,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c630b1709e678e0b00c070c7cac86d30; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72623327, jitterRate=0.08217190206050873}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:39,171 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1a4676c43c8662115dfd922e9ff72451; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64155009, jitterRate=-0.04401586949825287}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:39,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f1fc9bf98611ec0266597565431cc2a1; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70340890, jitterRate=0.048160940408706665}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:39,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1a4676c43c8662115dfd922e9ff72451: Running coprocessor pre-open hook at 1676145639160Writing region info on filesystem at 1676145639160Initializing all the Stores at 1676145639161 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145639161Cleaning up temporary data from old regions at 1676145639168 (+7 ms)Cleaning up detritus from prior splits at 1676145639168Running coprocessor post-open hooks at 1676145639171 (+3 ms)Region opened successfully at 1676145639171 2023-02-11 20:00:39,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f1fc9bf98611ec0266597565431cc2a1: Running coprocessor pre-open hook at 1676145639160Writing region info on filesystem at 1676145639160Initializing all the Stores at 1676145639160Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145639160Cleaning up temporary data from old regions at 1676145639168 (+8 ms)Cleaning up detritus from prior splits at 1676145639168Running coprocessor post-open hooks at 1676145639171 (+3 ms)Region opened successfully at 1676145639171 2023-02-11 20:00:39,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c630b1709e678e0b00c070c7cac86d30: Running coprocessor pre-open hook at 1676145639161Writing region info on filesystem at 1676145639161Initializing all the Stores at 1676145639161Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145639161Cleaning up temporary data from old regions at 1676145639168 (+7 ms)Cleaning up detritus from prior splits at 1676145639168Running coprocessor post-open hooks at 1676145639171 (+3 ms)Region opened successfully at 1676145639171 2023-02-11 20:00:39,171 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30., pid=130, masterSystemTime=1676145639158 2023-02-11 20:00:39,172 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1., pid=132, masterSystemTime=1676145639156 2023-02-11 20:00:39,172 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451., pid=128, masterSystemTime=1676145639156 2023-02-11 20:00:39,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:39,173 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:39,173 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:39,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 20:00:39,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:39,173 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:39,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:39,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:39,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:39,174 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639173"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145639173"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145639173"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145639173"}]},"ts":"1676145639173"} 2023-02-11 20:00:39,174 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:39,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:39,174 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:39,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 20:00:39,174 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:39,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:39,174 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639174"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145639174"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145639174"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145639174"}]},"ts":"1676145639174"} 2023-02-11 20:00:39,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:39,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:39,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:39,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:39,175 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:39,175 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:39,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 20:00:39,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:39,175 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:39,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:39,175 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145639175"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145639175"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145639175"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145639175"}]},"ts":"1676145639175"} 2023-02-11 20:00:39,176 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:39,176 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:39,176 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:39,176 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:39,178 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=124 2023-02-11 20:00:39,178 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:39,178 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=124, state=SUCCESS; OpenRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,44175,1676145619924 in 170 msec 2023-02-11 20:00:39,178 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf 2023-02-11 20:00:39,178 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf 2023-02-11 20:00:39,179 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region db772e5c38666094ec2e3b8fb5e805c2 columnFamilyName cf 2023-02-11 20:00:39,179 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf 2023-02-11 20:00:39,179 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf 2023-02-11 20:00:39,179 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=122 2023-02-11 20:00:39,179 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN in 336 msec 2023-02-11 20:00:39,180 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8de911ca888e28adfca434ff163cbba9 columnFamilyName cf 2023-02-11 20:00:39,180 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=122, state=SUCCESS; OpenRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887 in 166 msec 2023-02-11 20:00:39,180 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=126 2023-02-11 20:00:39,180 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(310): Store=db772e5c38666094ec2e3b8fb5e805c2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:39,180 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=126, state=SUCCESS; OpenRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,42011,1676145619841 in 174 msec 2023-02-11 20:00:39,180 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf 2023-02-11 20:00:39,180 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf 2023-02-11 20:00:39,181 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7c040ebb400ff6a8415e55ad85bbbe0e columnFamilyName cf 2023-02-11 20:00:39,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:39,181 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN in 338 msec 2023-02-11 20:00:39,182 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(310): Store=8de911ca888e28adfca434ff163cbba9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:39,182 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN in 338 msec 2023-02-11 20:00:39,182 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(310): Store=7c040ebb400ff6a8415e55ad85bbbe0e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:39,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:39,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:39,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:39,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:39,184 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:39,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:39,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:39,186 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened db772e5c38666094ec2e3b8fb5e805c2; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73273409, jitterRate=0.0918588787317276}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:39,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:39,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for db772e5c38666094ec2e3b8fb5e805c2: Running coprocessor pre-open hook at 1676145639174Writing region info on filesystem at 1676145639174Initializing all the Stores at 1676145639175 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145639175Cleaning up temporary data from old regions at 1676145639183 (+8 ms)Cleaning up detritus from prior splits at 1676145639183Running coprocessor post-open hooks at 1676145639186 (+3 ms)Region opened successfully at 1676145639186 2023-02-11 20:00:39,186 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2., pid=131, masterSystemTime=1676145639158 2023-02-11 20:00:39,186 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7c040ebb400ff6a8415e55ad85bbbe0e; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60121125, jitterRate=-0.10412542521953583}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:39,186 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 8de911ca888e28adfca434ff163cbba9; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59270413, jitterRate=-0.11680202186107635}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:39,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Running coprocessor pre-open hook at 1676145639176Writing region info on filesystem at 1676145639176Initializing all the Stores at 1676145639177 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145639178 (+1 ms)Cleaning up temporary data from old regions at 1676145639184 (+6 ms)Cleaning up detritus from prior splits at 1676145639184Running coprocessor post-open hooks at 1676145639186 (+2 ms)Region opened successfully at 1676145639186 2023-02-11 20:00:39,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 8de911ca888e28adfca434ff163cbba9: Running coprocessor pre-open hook at 1676145639175Writing region info on filesystem at 1676145639175Initializing all the Stores at 1676145639176 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145639176Cleaning up temporary data from old regions at 1676145639184 (+8 ms)Cleaning up detritus from prior splits at 1676145639184Running coprocessor post-open hooks at 1676145639187 (+3 ms)Region opened successfully at 1676145639187 2023-02-11 20:00:39,187 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e., pid=129, masterSystemTime=1676145639156 2023-02-11 20:00:39,187 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9., pid=127, masterSystemTime=1676145639156 2023-02-11 20:00:39,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:39,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:39,188 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:39,188 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145639188"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145639188"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145639188"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145639188"}]},"ts":"1676145639188"} 2023-02-11 20:00:39,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:39,189 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:39,189 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:39,189 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639189"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145639189"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145639189"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145639189"}]},"ts":"1676145639189"} 2023-02-11 20:00:39,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:39,190 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:39,190 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:39,190 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639190"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145639190"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145639190"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145639190"}]},"ts":"1676145639190"} 2023-02-11 20:00:39,200 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=121 2023-02-11 20:00:39,200 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=121, state=SUCCESS; OpenRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,44175,1676145619924 in 182 msec 2023-02-11 20:00:39,200 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=125 2023-02-11 20:00:39,200 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=125, state=SUCCESS; OpenRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,35937,1676145619887 in 189 msec 2023-02-11 20:00:39,200 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=123 2023-02-11 20:00:39,201 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=123, state=SUCCESS; OpenRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,42011,1676145619841 in 187 msec 2023-02-11 20:00:39,201 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN in 358 msec 2023-02-11 20:00:39,202 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN in 359 msec 2023-02-11 20:00:39,202 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=123, resume processing ppid=120 2023-02-11 20:00:39,202 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN in 358 msec 2023-02-11 20:00:39,203 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145639203"}]},"ts":"1676145639203"} 2023-02-11 20:00:39,205 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=ENABLED in hbase:meta 2023-02-11 20:00:39,207 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676145622835' was successfully enabled. 2023-02-11 20:00:39,208 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676145622835 in 373 msec 2023-02-11 20:00:39,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-11 20:00:39,590 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676145622835, procId: 120 completed 2023-02-11 20:00:39,590 DEBUG [Listener at localhost/33691] zookeeper.ReadOnlyZKClient(139): Connect 0x43ec0260 to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:39,595 DEBUG [Listener at localhost/33691] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54f3b092, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:39,597 DEBUG [hconnection-0x58227bee-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:39,603 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43680, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:39,606 DEBUG [hconnection-0x58227bee-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:39,608 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58568, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:39,609 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 20:00:39,617 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 20:00:39,620 DEBUG [hconnection-0x58227bee-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:39,621 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53826, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:39,622 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 20:00:39,627 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 20:00:39,632 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 20:00:39,638 INFO [Listener at localhost/33691] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676145622835 2023-02-11 20:00:39,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676145622835 2023-02-11 20:00:39,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:39,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-11 20:00:39,643 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145639643"}]},"ts":"1676145639643"} 2023-02-11 20:00:39,645 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=DISABLING in hbase:meta 2023-02-11 20:00:39,646 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676145622835 to state=DISABLING 2023-02-11 20:00:39,647 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN}] 2023-02-11 20:00:39,649 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN 2023-02-11 20:00:39,649 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN 2023-02-11 20:00:39,649 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN 2023-02-11 20:00:39,649 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN 2023-02-11 20:00:39,649 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN 2023-02-11 20:00:39,650 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:39,650 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:39,650 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639650"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145639650"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145639650"}]},"ts":"1676145639650"} 2023-02-11 20:00:39,650 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145639650"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145639650"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145639650"}]},"ts":"1676145639650"} 2023-02-11 20:00:39,650 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:39,650 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639650"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145639650"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145639650"}]},"ts":"1676145639650"} 2023-02-11 20:00:39,651 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:39,651 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145639651"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145639651"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145639651"}]},"ts":"1676145639651"} 2023-02-11 20:00:39,651 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:39,651 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639651"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145639651"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145639651"}]},"ts":"1676145639651"} 2023-02-11 20:00:39,652 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=136, state=RUNNABLE; CloseRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:39,652 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=134, state=RUNNABLE; CloseRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:39,655 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN 2023-02-11 20:00:39,656 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:39,656 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639656"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145639656"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145639656"}]},"ts":"1676145639656"} 2023-02-11 20:00:39,657 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=137, state=RUNNABLE; CloseRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:39,657 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=139, state=RUNNABLE; CloseRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:39,660 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=135, state=RUNNABLE; CloseRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:39,661 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=138, state=RUNNABLE; CloseRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:39,739 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 20:00:39,805 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:39,805 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:39,806 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7c040ebb400ff6a8415e55ad85bbbe0e, disabling compactions & flushes 2023-02-11 20:00:39,806 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c630b1709e678e0b00c070c7cac86d30, disabling compactions & flushes 2023-02-11 20:00:39,806 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:39,806 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:39,806 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:39,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. after waiting 0 ms 2023-02-11 20:00:39,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:39,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:39,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. after waiting 0 ms 2023-02-11 20:00:39,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:39,814 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:39,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 8de911ca888e28adfca434ff163cbba9, disabling compactions & flushes 2023-02-11 20:00:39,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:39,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:39,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. after waiting 0 ms 2023-02-11 20:00:39,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:39,820 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 20:00:39,821 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 20:00:39,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:39,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c630b1709e678e0b00c070c7cac86d30: Waiting for close lock at 1676145639806Running coprocessor pre-close hooks at 1676145639806Disabling compacts and flushes for region at 1676145639806Disabling writes for close at 1676145639807 (+1 ms)Writing region close event to WAL at 1676145639807Running coprocessor post-close hooks at 1676145639821 (+14 ms)Closed at 1676145639822 (+1 ms) 2023-02-11 20:00:39,823 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:39,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Waiting for close lock at 1676145639805Running coprocessor pre-close hooks at 1676145639805Disabling compacts and flushes for region at 1676145639805Disabling writes for close at 1676145639807 (+2 ms)Writing region close event to WAL at 1676145639807Running coprocessor post-close hooks at 1676145639822 (+15 ms)Closed at 1676145639823 (+1 ms) 2023-02-11 20:00:39,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 20:00:39,829 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:39,830 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 8de911ca888e28adfca434ff163cbba9: Waiting for close lock at 1676145639814Running coprocessor pre-close hooks at 1676145639814Disabling compacts and flushes for region at 1676145639814Disabling writes for close at 1676145639816 (+2 ms)Writing region close event to WAL at 1676145639820 (+4 ms)Running coprocessor post-close hooks at 1676145639829 (+9 ms)Closed at 1676145639829 2023-02-11 20:00:39,833 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:39,833 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:39,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing db772e5c38666094ec2e3b8fb5e805c2, disabling compactions & flushes 2023-02-11 20:00:39,833 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:39,834 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:39,834 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. after waiting 0 ms 2023-02-11 20:00:39,834 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:39,835 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=CLOSED 2023-02-11 20:00:39,835 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639835"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145639835"}]},"ts":"1676145639835"} 2023-02-11 20:00:39,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:39,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:39,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1a4676c43c8662115dfd922e9ff72451, disabling compactions & flushes 2023-02-11 20:00:39,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:39,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:39,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. after waiting 0 ms 2023-02-11 20:00:39,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:39,842 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=CLOSED 2023-02-11 20:00:39,842 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639842"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145639842"}]},"ts":"1676145639842"} 2023-02-11 20:00:39,842 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:39,842 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:39,844 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=CLOSED 2023-02-11 20:00:39,844 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639844"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145639844"}]},"ts":"1676145639844"} 2023-02-11 20:00:39,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 20:00:39,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f1fc9bf98611ec0266597565431cc2a1, disabling compactions & flushes 2023-02-11 20:00:39,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:39,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:39,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. after waiting 0 ms 2023-02-11 20:00:39,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:39,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:39,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for db772e5c38666094ec2e3b8fb5e805c2: Waiting for close lock at 1676145639833Running coprocessor pre-close hooks at 1676145639833Disabling compacts and flushes for region at 1676145639833Disabling writes for close at 1676145639834 (+1 ms)Writing region close event to WAL at 1676145639834Running coprocessor post-close hooks at 1676145639849 (+15 ms)Closed at 1676145639851 (+2 ms) 2023-02-11 20:00:39,852 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=137 2023-02-11 20:00:39,853 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=137, state=SUCCESS; CloseRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,44175,1676145619924 in 186 msec 2023-02-11 20:00:39,853 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=136 2023-02-11 20:00:39,853 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=136, state=SUCCESS; CloseRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,42011,1676145619841 in 192 msec 2023-02-11 20:00:39,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:39,856 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN in 206 msec 2023-02-11 20:00:39,857 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=CLOSED 2023-02-11 20:00:39,857 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=138 2023-02-11 20:00:39,857 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145639857"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145639857"}]},"ts":"1676145639857"} 2023-02-11 20:00:39,857 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=138, state=SUCCESS; CloseRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,35937,1676145619887 in 191 msec 2023-02-11 20:00:39,857 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN in 206 msec 2023-02-11 20:00:39,859 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN in 210 msec 2023-02-11 20:00:39,862 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=134 2023-02-11 20:00:39,862 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=134, state=SUCCESS; CloseRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,44175,1676145619924 in 207 msec 2023-02-11 20:00:39,864 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN in 215 msec 2023-02-11 20:00:39,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 20:00:39,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 20:00:39,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:39,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1a4676c43c8662115dfd922e9ff72451: Waiting for close lock at 1676145639836Running coprocessor pre-close hooks at 1676145639836Disabling compacts and flushes for region at 1676145639836Disabling writes for close at 1676145639837 (+1 ms)Writing region close event to WAL at 1676145639850 (+13 ms)Running coprocessor post-close hooks at 1676145639869 (+19 ms)Closed at 1676145639870 (+1 ms) 2023-02-11 20:00:39,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:39,871 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f1fc9bf98611ec0266597565431cc2a1: Waiting for close lock at 1676145639842Running coprocessor pre-close hooks at 1676145639842Disabling compacts and flushes for region at 1676145639842Disabling writes for close at 1676145639851 (+9 ms)Writing region close event to WAL at 1676145639853 (+2 ms)Running coprocessor post-close hooks at 1676145639870 (+17 ms)Closed at 1676145639871 (+1 ms) 2023-02-11 20:00:39,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:39,874 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=CLOSED 2023-02-11 20:00:39,875 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145639874"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145639874"}]},"ts":"1676145639874"} 2023-02-11 20:00:39,875 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:39,876 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=CLOSED 2023-02-11 20:00:39,876 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145639876"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145639876"}]},"ts":"1676145639876"} 2023-02-11 20:00:39,878 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=139 2023-02-11 20:00:39,878 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=139, state=SUCCESS; CloseRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,42011,1676145619841 in 219 msec 2023-02-11 20:00:39,881 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=135 2023-02-11 20:00:39,881 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=135, state=SUCCESS; CloseRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,35937,1676145619887 in 217 msec 2023-02-11 20:00:39,882 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN in 231 msec 2023-02-11 20:00:39,883 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=135, resume processing ppid=133 2023-02-11 20:00:39,883 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN in 234 msec 2023-02-11 20:00:39,884 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145639884"}]},"ts":"1676145639884"} 2023-02-11 20:00:39,885 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=DISABLED in hbase:meta 2023-02-11 20:00:39,891 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676145622835 to state=DISABLED 2023-02-11 20:00:39,893 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676145622835 in 254 msec 2023-02-11 20:00:39,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-11 20:00:39,894 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676145622835, procId: 133 completed 2023-02-11 20:00:39,903 INFO [Listener at localhost/33691] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1676145622835-1676145639903 2023-02-11 20:00:39,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb1-1676145622835-1676145639903 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:39,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676145639904 (current time:1676145639904). 2023-02-11 20:00:39,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 20:00:39,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(139): Connect 0x286c7cac to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:39,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@291e7746, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:39,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:39,919 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43694, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:39,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x286c7cac to 127.0.0.1:51732 2023-02-11 20:00:39,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:00:39,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 20:00:39,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1676145622835-1676145639903 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:39,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:39,924 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:39,925 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE; timeout=600000, timestamp=1676146239925 2023-02-11 20:00:39,925 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1676145622835-1676145639903 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676145622835 2023-02-11 20:00:39,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1676145622835-1676145639903 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } 2023-02-11 20:00:39,925 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1676145622835-1676145639903 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 20:00:39,925 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 20:00:39,951 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676145622835-1676145639903 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } 2023-02-11 20:00:39,959 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:39,959 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:39,959 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:39,959 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:39,959 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:39,959 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:39,960 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:39,960 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:39,960 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:39,960 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:39,960 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:39,960 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:39,960 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:39,960 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:39,962 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:39,962 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:39,967 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:39,967 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:39,967 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:39,968 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:39,975 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:40,175 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 20:00:40,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676145622835-1676145639903 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } is done 2023-02-11 20:00:40,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snaptb1-1676145622835-1676145639903 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 }' is still in progress! 2023-02-11 20:00:40,177 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-11 20:00:40,444 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1676145622835-1676145639903 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } as finished. 2023-02-11 20:00:40,444 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1676145622835-1676145639903 2023-02-11 20:00:40,446 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676145622835-1676145639903 2023-02-11 20:00:40,512 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676145622835-1676145639903 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/hbase-failsafe-snaptb1-1676145622835-1676145639903 2023-02-11 20:00:40,540 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1676145622835-1676145639903 of table testRestoreSnapshot-1676145622835 completed 2023-02-11 20:00:40,540 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676145622835-1676145639903 2023-02-11 20:00:40,541 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676145622835-1676145639903 2023-02-11 20:00:40,541 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1676145622835-1676145639903 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676145622835 at 1676145639925Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676145622835-1676145639903 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } at 1676145639951 (+26 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1676145622835-1676145639903 at 1676145640444 (+493 ms)Loading Region manifests for hbase-failsafe-snaptb1-1676145622835-1676145639903 at 1676145640445 (+1 ms)Writing data manifest for hbase-failsafe-snaptb1-1676145622835-1676145639903 at 1676145640486 (+41 ms)Verifying snapshot: hbase-failsafe-snaptb1-1676145622835-1676145639903 at 1676145640502 (+16 ms)Snapshot hbase-failsafe-snaptb1-1676145622835-1676145639903 of table testRestoreSnapshot-1676145622835 completed at 1676145640540 (+38 ms) 2023-02-11 20:00:40,542 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE 2023-02-11 20:00:40,544 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676145622835, type=EXCLUSIVE in 620 msec 2023-02-11 20:00:40,677 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 20:00:40,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676145622835-1676145639903 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 } is done 2023-02-11 20:00:40,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1676145622835-1676145639903 table=testRestoreSnapshot-1676145622835 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 20:00:40,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676145622835 snapshot=name: "snaptb1-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145634760 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 20:00:40,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1676145622835 as table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-11 20:00:40,718 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/.tabledesc/.tableinfo.0000000004 2023-02-11 20:00:40,719 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/.tabledesc/.tableinfo.0000000003 2023-02-11 20:00:40,720 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/.tabledesc/.tableinfo.0000000004 2023-02-11 20:00:40,721 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1676145622835 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } 2023-02-11 20:00:40,724 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145634760 type: DISABLED version: 2 ttl: 0 2023-02-11 20:00:40,724 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,733 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,735 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 of snapshot=snaptb1-1676145622835 to region=7cbf443d8efa7267c2d468f6e0b1360a table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,739 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de of snapshot=snaptb1-1676145622835 to region=7cbf443d8efa7267c2d468f6e0b1360a table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,743 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 of snapshot=snaptb1-1676145622835 to region=7cbf443d8efa7267c2d468f6e0b1360a table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,748 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 of snapshot=snaptb1-1676145622835 to region=7cbf443d8efa7267c2d468f6e0b1360a table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,753 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 of snapshot=snaptb1-1676145622835 to region=7cbf443d8efa7267c2d468f6e0b1360a table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,762 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 of snapshot=snaptb1-1676145622835 to region=7cbf443d8efa7267c2d468f6e0b1360a table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,766 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 of snapshot=snaptb1-1676145622835 to region=7cbf443d8efa7267c2d468f6e0b1360a table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,770 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d of snapshot=snaptb1-1676145622835 to region=7cbf443d8efa7267c2d468f6e0b1360a table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,773 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f of snapshot=snaptb1-1676145622835 to region=7cbf443d8efa7267c2d468f6e0b1360a table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,776 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e of snapshot=snaptb1-1676145622835 to region=7cbf443d8efa7267c2d468f6e0b1360a table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,789 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f of snapshot=snaptb1-1676145622835 to region=7cbf443d8efa7267c2d468f6e0b1360a table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,806 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 of snapshot=snaptb1-1676145622835 to region=7cbf443d8efa7267c2d468f6e0b1360a table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,817 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:40,817 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:40,817 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:40,817 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:40,817 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:40,817 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:40,841 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 233710642ffe449384153b4775e7e690 of snapshot=snaptb1-1676145622835 to region=7c040ebb400ff6a8415e55ad85bbbe0e table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,841 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 5eb756b4ee3a4959a80c36b5b304212c of snapshot=snaptb1-1676145622835 to region=c630b1709e678e0b00c070c7cac86d30 table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,843 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 85a30970077e4a559c5f7e7c79bd9336 of snapshot=snaptb1-1676145622835 to region=db772e5c38666094ec2e3b8fb5e805c2 table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,843 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 0bcc1b49b65f42d389fcb662a909dbf1 of snapshot=snaptb1-1676145622835 to region=1a4676c43c8662115dfd922e9ff72451 table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,848 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d185ce9140d04e959efc0b58ace714af of snapshot=snaptb1-1676145622835 to region=8de911ca888e28adfca434ff163cbba9 table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,849 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c116febc28fc423781baa2c67a3e1d2f of snapshot=snaptb1-1676145622835 to region=f1fc9bf98611ec0266597565431cc2a1 table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,859 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink cf1d617c62bd452786cb06efe103dd00 of snapshot=snaptb1-1676145622835 to region=c630b1709e678e0b00c070c7cac86d30 table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,860 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 2e4307f987394952952cf7c59270b4e7 of snapshot=snaptb1-1676145622835 to region=7c040ebb400ff6a8415e55ad85bbbe0e table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,873 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink cda13ed20aa14162b1be58c678d7c72c of snapshot=snaptb1-1676145622835 to region=1a4676c43c8662115dfd922e9ff72451 table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,876 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e793845eb1f447169fd725e5a836da1b of snapshot=snaptb1-1676145622835 to region=db772e5c38666094ec2e3b8fb5e805c2 table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,878 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e8a8517d5b3949c39858b0093dcf6481 of snapshot=snaptb1-1676145622835 to region=8de911ca888e28adfca434ff163cbba9 table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,891 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink f1723f83b8234b60b4690710a470498b of snapshot=snaptb1-1676145622835 to region=f1fc9bf98611ec0266597565431cc2a1 table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,899 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145634760 type: DISABLED version: 2 ttl: 0 2023-02-11 20:00:40,901 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145640900"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:40,901 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145640900"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:40,901 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145640900"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:40,901 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145640900"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:40,901 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145640900"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:40,901 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145640900"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:40,903 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 20:00:40,903 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-11 20:00:40,903 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145640901"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145640901"}]},"ts":"1676145640901"} 2023-02-11 20:00:40,903 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145640901"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145640901"}]},"ts":"1676145640901"} 2023-02-11 20:00:40,903 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145640901"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145640901"}]},"ts":"1676145640901"} 2023-02-11 20:00:40,903 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145640901"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145640901"}]},"ts":"1676145640901"} 2023-02-11 20:00:40,903 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145640901"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145640901"}]},"ts":"1676145640901"} 2023-02-11 20:00:40,903 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145640901"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145640901"}]},"ts":"1676145640901"} 2023-02-11 20:00:40,905 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 20:00:40,905 INFO [PEWorker-2] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-11 20:00:40,905 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-11 20:00:40,905 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1676145622835 table=testRestoreSnapshot-1676145622835 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676145622835 completed! 2023-02-11 20:00:40,907 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676145622835 snapshot=name: "snaptb1-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145634760 type: DISABLED version: 2 ttl: 0 ) in 209 msec 2023-02-11 20:00:40,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-11 20:00:40,955 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676145622835, procId: 147 completed 2023-02-11 20:00:40,955 INFO [Listener at localhost/33691] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1676145622835-1676145639903 2023-02-11 20:00:40,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb1-1676145622835-1676145639903" 2023-02-11 20:00:40,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1676145622835-1676145639903 2023-02-11 20:00:40,961 INFO [Listener at localhost/33691] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-11 20:00:40,966 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676145622835 2023-02-11 20:00:40,966 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145640966"}]},"ts":"1676145640966"} 2023-02-11 20:00:40,967 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=ENABLING in hbase:meta 2023-02-11 20:00:40,969 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 20:00:40,970 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN}] 2023-02-11 20:00:40,973 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN 2023-02-11 20:00:40,973 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN 2023-02-11 20:00:40,973 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN 2023-02-11 20:00:40,974 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN 2023-02-11 20:00:40,974 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:40,974 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:40,974 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:40,975 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:40,975 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN 2023-02-11 20:00:40,975 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN 2023-02-11 20:00:40,977 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:40,977 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 20:00:41,125 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 20:00:41,125 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 20:00:41,125 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 20:00:41,125 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 20:00:41,125 DEBUG [jenkins-hbase4:39971] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 20:00:41,129 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:41,129 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145641129"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145641129"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145641129"}]},"ts":"1676145641129"} 2023-02-11 20:00:41,129 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:41,130 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145641129"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145641129"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145641129"}]},"ts":"1676145641129"} 2023-02-11 20:00:41,130 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:41,130 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145641130"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145641130"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145641130"}]},"ts":"1676145641130"} 2023-02-11 20:00:41,130 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:41,130 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145641130"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145641130"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145641130"}]},"ts":"1676145641130"} 2023-02-11 20:00:41,131 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:41,131 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145641131"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145641131"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145641131"}]},"ts":"1676145641131"} 2023-02-11 20:00:41,133 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=153, state=RUNNABLE; OpenRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:41,135 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=150, state=RUNNABLE; OpenRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:41,136 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=152, state=RUNNABLE; OpenRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:41,137 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=149, state=RUNNABLE; OpenRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:41,138 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=154, state=RUNNABLE; OpenRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:41,138 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:41,138 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145641138"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145641138"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145641138"}]},"ts":"1676145641138"} 2023-02-11 20:00:41,141 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=151, state=RUNNABLE; OpenRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:41,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-11 20:00:41,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:41,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 20:00:41,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:41,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:41,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:41,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:41,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:41,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 20:00:41,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:41,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:41,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:41,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:41,291 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:41,291 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:41,292 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf 2023-02-11 20:00:41,292 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf 2023-02-11 20:00:41,292 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf 2023-02-11 20:00:41,292 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf 2023-02-11 20:00:41,292 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c630b1709e678e0b00c070c7cac86d30 columnFamilyName cf 2023-02-11 20:00:41,292 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f1fc9bf98611ec0266597565431cc2a1 columnFamilyName cf 2023-02-11 20:00:41,292 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:41,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 20:00:41,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:41,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:41,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:41,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:41,294 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:41,295 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf 2023-02-11 20:00:41,295 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf 2023-02-11 20:00:41,296 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a4676c43c8662115dfd922e9ff72451 columnFamilyName cf 2023-02-11 20:00:41,311 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-5eb756b4ee3a4959a80c36b5b304212c 2023-02-11 20:00:41,311 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-c116febc28fc423781baa2c67a3e1d2f 2023-02-11 20:00:41,311 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-0bcc1b49b65f42d389fcb662a909dbf1 2023-02-11 20:00:41,320 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-cda13ed20aa14162b1be58c678d7c72c 2023-02-11 20:00:41,320 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(310): Store=1a4676c43c8662115dfd922e9ff72451/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:41,321 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:41,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:41,324 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-f1723f83b8234b60b4690710a470498b 2023-02-11 20:00:41,324 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(310): Store=f1fc9bf98611ec0266597565431cc2a1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:41,325 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-cf1d617c62bd452786cb06efe103dd00 2023-02-11 20:00:41,325 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(310): Store=c630b1709e678e0b00c070c7cac86d30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:41,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:41,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:41,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:41,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:41,329 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1a4676c43c8662115dfd922e9ff72451; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67443029, jitterRate=0.004979446530342102}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:41,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1a4676c43c8662115dfd922e9ff72451: Running coprocessor pre-open hook at 1676145641293Writing region info on filesystem at 1676145641293Initializing all the Stores at 1676145641294 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145641294Cleaning up temporary data from old regions at 1676145641323 (+29 ms)Cleaning up detritus from prior splits at 1676145641325 (+2 ms)Running coprocessor post-open hooks at 1676145641329 (+4 ms)Region opened successfully at 1676145641329 2023-02-11 20:00:41,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:41,332 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451., pid=159, masterSystemTime=1676145641290 2023-02-11 20:00:41,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:41,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:41,333 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:41,334 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:41,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 20:00:41,334 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:41,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:41,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:41,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:41,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:41,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:41,334 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145641334"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145641334"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145641334"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145641334"}]},"ts":"1676145641334"} 2023-02-11 20:00:41,334 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c630b1709e678e0b00c070c7cac86d30; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68981875, jitterRate=0.027910038828849792}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:41,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c630b1709e678e0b00c070c7cac86d30: Running coprocessor pre-open hook at 1676145641290Writing region info on filesystem at 1676145641290Initializing all the Stores at 1676145641291 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145641291Cleaning up temporary data from old regions at 1676145641328 (+37 ms)Cleaning up detritus from prior splits at 1676145641329 (+1 ms)Running coprocessor post-open hooks at 1676145641335 (+6 ms)Region opened successfully at 1676145641335 2023-02-11 20:00:41,336 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30., pid=157, masterSystemTime=1676145641287 2023-02-11 20:00:41,336 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f1fc9bf98611ec0266597565431cc2a1; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66582839, jitterRate=-0.007838383316993713}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:41,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f1fc9bf98611ec0266597565431cc2a1: Running coprocessor pre-open hook at 1676145641290Writing region info on filesystem at 1676145641290Initializing all the Stores at 1676145641290Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145641290Cleaning up temporary data from old regions at 1676145641330 (+40 ms)Cleaning up detritus from prior splits at 1676145641331 (+1 ms)Running coprocessor post-open hooks at 1676145641336 (+5 ms)Region opened successfully at 1676145641336 2023-02-11 20:00:41,337 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1., pid=156, masterSystemTime=1676145641286 2023-02-11 20:00:41,338 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:41,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:41,338 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:41,338 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:41,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 20:00:41,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:41,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:41,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:41,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:41,340 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:41,340 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145641340"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145641340"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145641340"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145641340"}]},"ts":"1676145641340"} 2023-02-11 20:00:41,341 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=154 2023-02-11 20:00:41,341 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=154, state=SUCCESS; OpenRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,35937,1676145619887 in 199 msec 2023-02-11 20:00:41,341 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:41,341 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:41,341 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145641341"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145641341"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145641341"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145641341"}]},"ts":"1676145641341"} 2023-02-11 20:00:41,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:41,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:41,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:41,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 20:00:41,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:41,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:41,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:41,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:41,342 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf 2023-02-11 20:00:41,342 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf 2023-02-11 20:00:41,344 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf 2023-02-11 20:00:41,344 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf 2023-02-11 20:00:41,344 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7c040ebb400ff6a8415e55ad85bbbe0e columnFamilyName cf 2023-02-11 20:00:41,344 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN in 371 msec 2023-02-11 20:00:41,345 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=152 2023-02-11 20:00:41,345 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=152, state=SUCCESS; OpenRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,42011,1676145619841 in 206 msec 2023-02-11 20:00:41,346 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=150 2023-02-11 20:00:41,346 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=150, state=SUCCESS; OpenRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,44175,1676145619924 in 208 msec 2023-02-11 20:00:41,347 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN in 375 msec 2023-02-11 20:00:41,347 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN in 376 msec 2023-02-11 20:00:41,348 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region db772e5c38666094ec2e3b8fb5e805c2 columnFamilyName cf 2023-02-11 20:00:41,352 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:41,354 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf 2023-02-11 20:00:41,354 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf 2023-02-11 20:00:41,354 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8de911ca888e28adfca434ff163cbba9 columnFamilyName cf 2023-02-11 20:00:41,365 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-233710642ffe449384153b4775e7e690 2023-02-11 20:00:41,367 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-85a30970077e4a559c5f7e7c79bd9336 2023-02-11 20:00:41,374 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-d185ce9140d04e959efc0b58ace714af 2023-02-11 20:00:41,391 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-e793845eb1f447169fd725e5a836da1b 2023-02-11 20:00:41,394 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(310): Store=db772e5c38666094ec2e3b8fb5e805c2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:41,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:41,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:41,398 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-e8a8517d5b3949c39858b0093dcf6481 2023-02-11 20:00:41,399 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(310): Store=8de911ca888e28adfca434ff163cbba9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:41,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:41,400 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-2e4307f987394952952cf7c59270b4e7 2023-02-11 20:00:41,401 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(310): Store=7c040ebb400ff6a8415e55ad85bbbe0e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:41,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:41,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:41,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:41,403 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:41,403 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened db772e5c38666094ec2e3b8fb5e805c2; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63582207, jitterRate=-0.052551284432411194}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:41,403 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for db772e5c38666094ec2e3b8fb5e805c2: Running coprocessor pre-open hook at 1676145641339Writing region info on filesystem at 1676145641339Initializing all the Stores at 1676145641340 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145641340Cleaning up temporary data from old regions at 1676145641398 (+58 ms)Cleaning up detritus from prior splits at 1676145641399 (+1 ms)Running coprocessor post-open hooks at 1676145641403 (+4 ms)Region opened successfully at 1676145641403 2023-02-11 20:00:41,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:41,404 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2., pid=158, masterSystemTime=1676145641287 2023-02-11 20:00:41,405 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 8de911ca888e28adfca434ff163cbba9; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62347530, jitterRate=-0.07094940543174744}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:41,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 8de911ca888e28adfca434ff163cbba9: Running coprocessor pre-open hook at 1676145641342Writing region info on filesystem at 1676145641342Initializing all the Stores at 1676145641344 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145641344Cleaning up temporary data from old regions at 1676145641401 (+57 ms)Cleaning up detritus from prior splits at 1676145641402 (+1 ms)Running coprocessor post-open hooks at 1676145641405 (+3 ms)Region opened successfully at 1676145641405 2023-02-11 20:00:41,406 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9., pid=155, masterSystemTime=1676145641286 2023-02-11 20:00:41,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:41,406 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:41,407 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:41,407 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145641406"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145641406"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145641406"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145641406"}]},"ts":"1676145641406"} 2023-02-11 20:00:41,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:41,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:41,408 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:41,408 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:41,408 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145641408"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145641408"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145641408"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145641408"}]},"ts":"1676145641408"} 2023-02-11 20:00:41,408 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7c040ebb400ff6a8415e55ad85bbbe0e; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63588368, jitterRate=-0.0524594783782959}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:41,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Running coprocessor pre-open hook at 1676145641335Writing region info on filesystem at 1676145641335Initializing all the Stores at 1676145641336 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145641337 (+1 ms)Cleaning up temporary data from old regions at 1676145641404 (+67 ms)Cleaning up detritus from prior splits at 1676145641404Running coprocessor post-open hooks at 1676145641408 (+4 ms)Region opened successfully at 1676145641408 2023-02-11 20:00:41,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e., pid=160, masterSystemTime=1676145641290 2023-02-11 20:00:41,422 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=149 2023-02-11 20:00:41,422 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=149, state=SUCCESS; OpenRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841 in 271 msec 2023-02-11 20:00:41,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:41,423 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:41,423 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:41,423 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145641423"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145641423"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145641423"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145641423"}]},"ts":"1676145641423"} 2023-02-11 20:00:41,424 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=153 2023-02-11 20:00:41,424 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN in 452 msec 2023-02-11 20:00:41,424 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=153, state=SUCCESS; OpenRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,44175,1676145619924 in 288 msec 2023-02-11 20:00:41,425 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN in 454 msec 2023-02-11 20:00:41,427 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=151 2023-02-11 20:00:41,427 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=151, state=SUCCESS; OpenRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,35937,1676145619887 in 284 msec 2023-02-11 20:00:41,429 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=151, resume processing ppid=148 2023-02-11 20:00:41,429 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN in 457 msec 2023-02-11 20:00:41,430 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145641430"}]},"ts":"1676145641430"} 2023-02-11 20:00:41,431 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=ENABLED in hbase:meta 2023-02-11 20:00:41,433 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676145622835' was successfully enabled. 2023-02-11 20:00:41,435 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676145622835 in 473 msec 2023-02-11 20:00:41,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-11 20:00:41,717 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676145622835, procId: 148 completed 2023-02-11 20:00:41,718 DEBUG [Listener at localhost/33691] zookeeper.ReadOnlyZKClient(139): Connect 0x70b95fe7 to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:41,723 DEBUG [Listener at localhost/33691] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ad94238, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:41,725 DEBUG [hconnection-0x3a5b8d75-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:41,728 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43706, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:41,733 DEBUG [hconnection-0x3a5b8d75-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:41,738 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53840, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:41,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,866 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43ec0260 to 127.0.0.1:51732 2023-02-11 20:00:41,866 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:00:41,867 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x251148a9 to 127.0.0.1:51732 2023-02-11 20:00:41,867 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:00:41,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,976 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 20:00:41,980 DEBUG [hconnection-0x3a5b8d75-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:41,984 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58570, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:41,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:41,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,188 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 20:00:42,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,513 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 20:00:42,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,697 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 20:00:42,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,879 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 20:00:42,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:42,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:43,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,664 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 20:00:44,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:44,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:45,443 INFO [Listener at localhost/33691] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676145622835 2023-02-11 20:00:45,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676145622835 2023-02-11 20:00:45,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:45,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-11 20:00:45,448 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145645448"}]},"ts":"1676145645448"} 2023-02-11 20:00:45,449 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=DISABLING in hbase:meta 2023-02-11 20:00:45,451 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676145622835 to state=DISABLING 2023-02-11 20:00:45,451 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN}] 2023-02-11 20:00:45,453 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN 2023-02-11 20:00:45,453 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN 2023-02-11 20:00:45,453 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN 2023-02-11 20:00:45,453 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN 2023-02-11 20:00:45,454 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN 2023-02-11 20:00:45,454 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:45,454 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145645454"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145645454"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145645454"}]},"ts":"1676145645454"} 2023-02-11 20:00:45,454 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:45,454 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145645454"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145645454"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145645454"}]},"ts":"1676145645454"} 2023-02-11 20:00:45,455 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:45,455 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145645455"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145645455"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145645455"}]},"ts":"1676145645455"} 2023-02-11 20:00:45,454 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:45,455 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145645454"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145645454"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145645454"}]},"ts":"1676145645454"} 2023-02-11 20:00:45,455 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:45,455 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=165, state=RUNNABLE; CloseRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:45,455 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145645455"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145645455"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145645455"}]},"ts":"1676145645455"} 2023-02-11 20:00:45,456 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=163, state=RUNNABLE; CloseRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:45,456 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=162, state=RUNNABLE; CloseRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:45,458 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=164, state=RUNNABLE; CloseRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:45,458 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=167, state=RUNNABLE; CloseRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:45,459 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN 2023-02-11 20:00:45,460 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:45,460 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145645460"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145645460"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145645460"}]},"ts":"1676145645460"} 2023-02-11 20:00:45,462 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=166, state=RUNNABLE; CloseRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:45,607 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:45,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c630b1709e678e0b00c070c7cac86d30, disabling compactions & flushes 2023-02-11 20:00:45,608 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:45,609 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:45,609 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. after waiting 0 ms 2023-02-11 20:00:45,609 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:45,609 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:45,610 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f1fc9bf98611ec0266597565431cc2a1, disabling compactions & flushes 2023-02-11 20:00:45,610 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:45,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:45,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. after waiting 0 ms 2023-02-11 20:00:45,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:45,613 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:45,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7c040ebb400ff6a8415e55ad85bbbe0e, disabling compactions & flushes 2023-02-11 20:00:45,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:45,615 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:45,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. after waiting 0 ms 2023-02-11 20:00:45,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:45,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 20:00:45,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 20:00:45,625 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 20:00:45,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:45,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c630b1709e678e0b00c070c7cac86d30: Waiting for close lock at 1676145645608Running coprocessor pre-close hooks at 1676145645608Disabling compacts and flushes for region at 1676145645608Disabling writes for close at 1676145645609 (+1 ms)Writing region close event to WAL at 1676145645613 (+4 ms)Running coprocessor post-close hooks at 1676145645625 (+12 ms)Closed at 1676145645625 2023-02-11 20:00:45,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:45,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f1fc9bf98611ec0266597565431cc2a1: Waiting for close lock at 1676145645609Running coprocessor pre-close hooks at 1676145645609Disabling compacts and flushes for region at 1676145645609Disabling writes for close at 1676145645613 (+4 ms)Writing region close event to WAL at 1676145645616 (+3 ms)Running coprocessor post-close hooks at 1676145645625 (+9 ms)Closed at 1676145645626 (+1 ms) 2023-02-11 20:00:45,628 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:45,628 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Waiting for close lock at 1676145645613Running coprocessor pre-close hooks at 1676145645613Disabling compacts and flushes for region at 1676145645613Disabling writes for close at 1676145645616 (+3 ms)Writing region close event to WAL at 1676145645619 (+3 ms)Running coprocessor post-close hooks at 1676145645626 (+7 ms)Closed at 1676145645628 (+2 ms) 2023-02-11 20:00:45,629 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:45,629 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:45,629 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing db772e5c38666094ec2e3b8fb5e805c2, disabling compactions & flushes 2023-02-11 20:00:45,629 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:45,629 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:45,629 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. after waiting 0 ms 2023-02-11 20:00:45,630 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:45,630 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=CLOSED 2023-02-11 20:00:45,630 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145645629"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145645629"}]},"ts":"1676145645629"} 2023-02-11 20:00:45,630 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:45,630 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:45,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 8de911ca888e28adfca434ff163cbba9, disabling compactions & flushes 2023-02-11 20:00:45,632 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:45,632 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:45,632 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. after waiting 0 ms 2023-02-11 20:00:45,632 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:45,634 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=CLOSED 2023-02-11 20:00:45,638 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145645634"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145645634"}]},"ts":"1676145645634"} 2023-02-11 20:00:45,638 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:45,639 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:45,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1a4676c43c8662115dfd922e9ff72451, disabling compactions & flushes 2023-02-11 20:00:45,639 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:45,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:45,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. after waiting 0 ms 2023-02-11 20:00:45,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:45,640 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=CLOSED 2023-02-11 20:00:45,640 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145645640"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145645640"}]},"ts":"1676145645640"} 2023-02-11 20:00:45,644 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 20:00:45,648 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=165 2023-02-11 20:00:45,648 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=165, state=SUCCESS; CloseRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,42011,1676145619841 in 183 msec 2023-02-11 20:00:45,648 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=163 2023-02-11 20:00:45,649 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=163, state=SUCCESS; CloseRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,44175,1676145619924 in 184 msec 2023-02-11 20:00:45,649 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=164 2023-02-11 20:00:45,649 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=164, state=SUCCESS; CloseRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,35937,1676145619887 in 186 msec 2023-02-11 20:00:45,651 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN in 197 msec 2023-02-11 20:00:45,652 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN in 198 msec 2023-02-11 20:00:45,652 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN in 198 msec 2023-02-11 20:00:45,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:45,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for db772e5c38666094ec2e3b8fb5e805c2: Waiting for close lock at 1676145645629Running coprocessor pre-close hooks at 1676145645629Disabling compacts and flushes for region at 1676145645629Disabling writes for close at 1676145645629Writing region close event to WAL at 1676145645634 (+5 ms)Running coprocessor post-close hooks at 1676145645653 (+19 ms)Closed at 1676145645653 2023-02-11 20:00:45,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 20:00:45,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:45,654 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 8de911ca888e28adfca434ff163cbba9: Waiting for close lock at 1676145645630Running coprocessor pre-close hooks at 1676145645630Disabling compacts and flushes for region at 1676145645630Disabling writes for close at 1676145645632 (+2 ms)Writing region close event to WAL at 1676145645638 (+6 ms)Running coprocessor post-close hooks at 1676145645654 (+16 ms)Closed at 1676145645654 2023-02-11 20:00:45,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:45,655 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=CLOSED 2023-02-11 20:00:45,655 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145645655"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145645655"}]},"ts":"1676145645655"} 2023-02-11 20:00:45,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:45,659 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=CLOSED 2023-02-11 20:00:45,660 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145645659"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145645659"}]},"ts":"1676145645659"} 2023-02-11 20:00:45,662 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=162 2023-02-11 20:00:45,662 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=162, state=SUCCESS; CloseRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841 in 204 msec 2023-02-11 20:00:45,662 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=166 2023-02-11 20:00:45,663 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=166, state=SUCCESS; CloseRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,44175,1676145619924 in 199 msec 2023-02-11 20:00:45,663 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN in 211 msec 2023-02-11 20:00:45,663 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN in 211 msec 2023-02-11 20:00:45,667 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 20:00:45,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:45,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1a4676c43c8662115dfd922e9ff72451: Waiting for close lock at 1676145645639Running coprocessor pre-close hooks at 1676145645639Disabling compacts and flushes for region at 1676145645639Disabling writes for close at 1676145645639Writing region close event to WAL at 1676145645644 (+5 ms)Running coprocessor post-close hooks at 1676145645668 (+24 ms)Closed at 1676145645668 2023-02-11 20:00:45,669 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:45,670 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=CLOSED 2023-02-11 20:00:45,670 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145645670"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145645670"}]},"ts":"1676145645670"} 2023-02-11 20:00:45,674 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=167 2023-02-11 20:00:45,674 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=167, state=SUCCESS; CloseRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,35937,1676145619887 in 213 msec 2023-02-11 20:00:45,676 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=167, resume processing ppid=161 2023-02-11 20:00:45,676 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN in 223 msec 2023-02-11 20:00:45,677 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145645677"}]},"ts":"1676145645677"} 2023-02-11 20:00:45,678 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=DISABLED in hbase:meta 2023-02-11 20:00:45,679 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676145622835 to state=DISABLED 2023-02-11 20:00:45,681 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676145622835 in 236 msec 2023-02-11 20:00:45,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-11 20:00:45,700 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676145622835, procId: 161 completed 2023-02-11 20:00:45,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676145622835 2023-02-11 20:00:45,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:45,715 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:45,717 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:45,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-11 20:00:45,726 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:45,726 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:45,726 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:45,726 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:45,726 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:45,726 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:45,729 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/recovered.edits] 2023-02-11 20:00:45,729 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/recovered.edits] 2023-02-11 20:00:45,729 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/recovered.edits] 2023-02-11 20:00:45,729 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/recovered.edits] 2023-02-11 20:00:45,729 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/recovered.edits] 2023-02-11 20:00:45,729 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/recovered.edits] 2023-02-11 20:00:45,732 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-233710642ffe449384153b4775e7e690 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-233710642ffe449384153b4775e7e690 2023-02-11 20:00:45,733 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-85a30970077e4a559c5f7e7c79bd9336 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-85a30970077e4a559c5f7e7c79bd9336 2023-02-11 20:00:45,733 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-0bcc1b49b65f42d389fcb662a909dbf1 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-0bcc1b49b65f42d389fcb662a909dbf1 2023-02-11 20:00:45,734 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-c116febc28fc423781baa2c67a3e1d2f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-c116febc28fc423781baa2c67a3e1d2f 2023-02-11 20:00:45,734 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-5eb756b4ee3a4959a80c36b5b304212c to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-5eb756b4ee3a4959a80c36b5b304212c 2023-02-11 20:00:45,734 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-2e4307f987394952952cf7c59270b4e7 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-2e4307f987394952952cf7c59270b4e7 2023-02-11 20:00:45,734 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-d185ce9140d04e959efc0b58ace714af to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-d185ce9140d04e959efc0b58ace714af 2023-02-11 20:00:45,735 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-e793845eb1f447169fd725e5a836da1b to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-e793845eb1f447169fd725e5a836da1b 2023-02-11 20:00:45,736 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-cda13ed20aa14162b1be58c678d7c72c to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-cda13ed20aa14162b1be58c678d7c72c 2023-02-11 20:00:45,736 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-f1723f83b8234b60b4690710a470498b to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-f1723f83b8234b60b4690710a470498b 2023-02-11 20:00:45,737 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-cf1d617c62bd452786cb06efe103dd00 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-cf1d617c62bd452786cb06efe103dd00 2023-02-11 20:00:45,737 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-e8a8517d5b3949c39858b0093dcf6481 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-e8a8517d5b3949c39858b0093dcf6481 2023-02-11 20:00:45,739 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/recovered.edits/27.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/recovered.edits/27.seqid 2023-02-11 20:00:45,740 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:45,741 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/recovered.edits/27.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/recovered.edits/27.seqid 2023-02-11 20:00:45,741 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/recovered.edits/27.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/recovered.edits/27.seqid 2023-02-11 20:00:45,742 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:45,742 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/recovered.edits/27.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/recovered.edits/27.seqid 2023-02-11 20:00:45,742 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:45,743 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:45,743 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/recovered.edits/27.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/recovered.edits/27.seqid 2023-02-11 20:00:45,744 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:45,744 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/recovered.edits/27.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/recovered.edits/27.seqid 2023-02-11 20:00:45,744 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:45,744 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676145622835 regions 2023-02-11 20:00:45,745 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a 2023-02-11 20:00:45,746 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf] 2023-02-11 20:00:45,748 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 2023-02-11 20:00:45,749 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de 2023-02-11 20:00:45,750 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 2023-02-11 20:00:45,751 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 2023-02-11 20:00:45,752 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 2023-02-11 20:00:45,753 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 2023-02-11 20:00:45,754 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 2023-02-11 20:00:45,755 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d 2023-02-11 20:00:45,756 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f 2023-02-11 20:00:45,757 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e 2023-02-11 20:00:45,758 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f 2023-02-11 20:00:45,759 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 2023-02-11 20:00:45,759 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a 2023-02-11 20:00:45,761 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:45,782 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676145622835 from hbase:meta 2023-02-11 20:00:45,784 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676145622835' descriptor. 2023-02-11 20:00:45,785 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:45,785 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676145622835' from region states. 2023-02-11 20:00:45,786 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145645785"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:45,786 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145645785"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:45,786 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145645785"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:45,786 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145645785"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:45,786 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145645785"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:45,786 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145645785"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:45,787 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 20:00:45,788 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''}] 2023-02-11 20:00:45,788 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676145622835' as deleted. 2023-02-11 20:00:45,788 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676145645788"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:45,789 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676145622835 state from META 2023-02-11 20:00:45,791 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:45,792 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676145622835 in 85 msec 2023-02-11 20:00:45,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-11 20:00:45,970 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676145622835, procId: 174 completed 2023-02-11 20:00:46,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1676145622835 snapshot=name: "snaptb1-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145634760 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 20:00:46,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1676145622835 as table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-11 20:00:46,021 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/.tabledesc/.tableinfo.0000000001 2023-02-11 20:00:46,023 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145634760 type: DISABLED version: 2 ttl: 0 2023-02-11 20:00:46,023 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,024 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,026 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,028 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,032 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,036 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,039 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,041 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,043 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,046 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,049 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,051 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,057 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,059 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:46,060 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:46,060 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:46,060 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:46,060 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:46,060 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:46,060 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=8de911ca888e28adfca434ff163cbba9 as 8de911ca888e28adfca434ff163cbba9 in snapshot snaptb1-1676145622835 2023-02-11 20:00:46,060 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=f1fc9bf98611ec0266597565431cc2a1 as f1fc9bf98611ec0266597565431cc2a1 in snapshot snaptb1-1676145622835 2023-02-11 20:00:46,060 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=1a4676c43c8662115dfd922e9ff72451 as 1a4676c43c8662115dfd922e9ff72451 in snapshot snaptb1-1676145622835 2023-02-11 20:00:46,060 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=db772e5c38666094ec2e3b8fb5e805c2 as db772e5c38666094ec2e3b8fb5e805c2 in snapshot snaptb1-1676145622835 2023-02-11 20:00:46,060 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=7c040ebb400ff6a8415e55ad85bbbe0e as 7c040ebb400ff6a8415e55ad85bbbe0e in snapshot snaptb1-1676145622835 2023-02-11 20:00:46,060 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=c630b1709e678e0b00c070c7cac86d30 as c630b1709e678e0b00c070c7cac86d30 in snapshot snaptb1-1676145622835 2023-02-11 20:00:46,061 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7671): creating {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676145622835', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:46,062 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7671): creating {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676145622835', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:46,062 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7671): creating {ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676145622835', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:46,062 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7671): creating {ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676145622835', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:46,062 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7671): creating {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676145622835', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:46,064 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7671): creating {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676145622835', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:46,094 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:46,094 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 5eb756b4ee3a4959a80c36b5b304212c from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,095 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:46,095 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 0bcc1b49b65f42d389fcb662a909dbf1 from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,098 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:46,098 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 233710642ffe449384153b4775e7e690 from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,099 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:46,099 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d185ce9140d04e959efc0b58ace714af from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,099 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink cf1d617c62bd452786cb06efe103dd00 from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,102 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink cda13ed20aa14162b1be58c678d7c72c from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,102 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 2e4307f987394952952cf7c59270b4e7 from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,102 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e8a8517d5b3949c39858b0093dcf6481 from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,103 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1603): Closing c630b1709e678e0b00c070c7cac86d30, disabling compactions & flushes 2023-02-11 20:00:46,103 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:46,103 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:46,103 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. after waiting 0 ms 2023-02-11 20:00:46,103 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:46,103 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:46,103 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1557): Region close journal for c630b1709e678e0b00c070c7cac86d30: Waiting for close lock at 1676145646103Disabling compacts and flushes for region at 1676145646103Disabling writes for close at 1676145646103Writing region close event to WAL at 1676145646103Closed at 1676145646103 2023-02-11 20:00:46,106 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1603): Closing 8de911ca888e28adfca434ff163cbba9, disabling compactions & flushes 2023-02-11 20:00:46,106 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:46,106 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1603): Closing 1a4676c43c8662115dfd922e9ff72451, disabling compactions & flushes 2023-02-11 20:00:46,106 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1603): Closing 7c040ebb400ff6a8415e55ad85bbbe0e, disabling compactions & flushes 2023-02-11 20:00:46,106 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:46,106 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:46,106 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:46,106 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. after waiting 0 ms 2023-02-11 20:00:46,106 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:46,106 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:46,106 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. after waiting 0 ms 2023-02-11 20:00:46,106 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:46,106 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:46,106 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:46,106 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1557): Region close journal for 8de911ca888e28adfca434ff163cbba9: Waiting for close lock at 1676145646106Disabling compacts and flushes for region at 1676145646106Disabling writes for close at 1676145646106Writing region close event to WAL at 1676145646106Closed at 1676145646106 2023-02-11 20:00:46,106 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:46,106 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. after waiting 0 ms 2023-02-11 20:00:46,106 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:46,106 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1557): Region close journal for 1a4676c43c8662115dfd922e9ff72451: Waiting for close lock at 1676145646106Disabling compacts and flushes for region at 1676145646106Disabling writes for close at 1676145646106Writing region close event to WAL at 1676145646106Closed at 1676145646106 2023-02-11 20:00:46,106 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:46,106 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1557): Region close journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Waiting for close lock at 1676145646106Disabling compacts and flushes for region at 1676145646106Disabling writes for close at 1676145646106Writing region close event to WAL at 1676145646106Closed at 1676145646106 2023-02-11 20:00:46,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-11 20:00:46,489 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:46,489 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c116febc28fc423781baa2c67a3e1d2f from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,492 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink f1723f83b8234b60b4690710a470498b from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,494 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1603): Closing f1fc9bf98611ec0266597565431cc2a1, disabling compactions & flushes 2023-02-11 20:00:46,495 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:46,495 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:46,495 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. after waiting 0 ms 2023-02-11 20:00:46,495 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:46,495 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:46,495 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1557): Region close journal for f1fc9bf98611ec0266597565431cc2a1: Waiting for close lock at 1676145646494Disabling compacts and flushes for region at 1676145646494Disabling writes for close at 1676145646495 (+1 ms)Writing region close event to WAL at 1676145646495Closed at 1676145646495 2023-02-11 20:00:46,499 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:46,499 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 85a30970077e4a559c5f7e7c79bd9336 from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,501 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e793845eb1f447169fd725e5a836da1b from cloned region in snapshot snaptb1-1676145622835 to table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:46,502 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1603): Closing db772e5c38666094ec2e3b8fb5e805c2, disabling compactions & flushes 2023-02-11 20:00:46,503 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:46,503 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:46,503 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. after waiting 0 ms 2023-02-11 20:00:46,503 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:46,503 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:46,503 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1557): Region close journal for db772e5c38666094ec2e3b8fb5e805c2: Waiting for close lock at 1676145646502Disabling compacts and flushes for region at 1676145646502Disabling writes for close at 1676145646503 (+1 ms)Writing region close event to WAL at 1676145646503Closed at 1676145646503 2023-02-11 20:00:46,503 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145634760 type: DISABLED version: 2 ttl: 0 2023-02-11 20:00:46,503 INFO [PEWorker-5] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1676145622835 on table=testRestoreSnapshot-1676145622835 completed! 2023-02-11 20:00:46,507 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145646507"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145646507"}]},"ts":"1676145646507"} 2023-02-11 20:00:46,507 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145646507"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145646507"}]},"ts":"1676145646507"} 2023-02-11 20:00:46,507 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145646507"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145646507"}]},"ts":"1676145646507"} 2023-02-11 20:00:46,507 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145646507"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145646507"}]},"ts":"1676145646507"} 2023-02-11 20:00:46,507 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145646507"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145646507"}]},"ts":"1676145646507"} 2023-02-11 20:00:46,508 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145646507"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145646507"}]},"ts":"1676145646507"} 2023-02-11 20:00:46,509 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 20:00:46,510 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145646510"}]},"ts":"1676145646510"} 2023-02-11 20:00:46,511 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=ENABLING in hbase:meta 2023-02-11 20:00:46,516 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 20:00:46,516 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 20:00:46,516 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 20:00:46,516 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 20:00:46,516 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 20:00:46,516 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN}] 2023-02-11 20:00:46,518 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN 2023-02-11 20:00:46,518 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN 2023-02-11 20:00:46,519 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN 2023-02-11 20:00:46,519 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN 2023-02-11 20:00:46,519 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44175,1676145619924; forceNewPlan=false, retain=false 2023-02-11 20:00:46,520 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42011,1676145619841; forceNewPlan=false, retain=false 2023-02-11 20:00:46,520 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44175,1676145619924; forceNewPlan=false, retain=false 2023-02-11 20:00:46,520 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN 2023-02-11 20:00:46,520 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35937,1676145619887; forceNewPlan=false, retain=false 2023-02-11 20:00:46,520 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN 2023-02-11 20:00:46,521 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35937,1676145619887; forceNewPlan=false, retain=false 2023-02-11 20:00:46,521 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42011,1676145619841; forceNewPlan=false, retain=false 2023-02-11 20:00:46,670 INFO [jenkins-hbase4:39971] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 20:00:46,673 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:46,673 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:46,673 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145646673"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145646673"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145646673"}]},"ts":"1676145646673"} 2023-02-11 20:00:46,673 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:46,673 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:46,673 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145646673"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145646673"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145646673"}]},"ts":"1676145646673"} 2023-02-11 20:00:46,673 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:46,673 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145646673"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145646673"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145646673"}]},"ts":"1676145646673"} 2023-02-11 20:00:46,674 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145646673"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145646673"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145646673"}]},"ts":"1676145646673"} 2023-02-11 20:00:46,673 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145646673"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145646673"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145646673"}]},"ts":"1676145646673"} 2023-02-11 20:00:46,675 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=176, state=RUNNABLE; OpenRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:46,677 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=179, state=RUNNABLE; OpenRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:46,681 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=180, state=RUNNABLE; OpenRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:46,682 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=177, state=RUNNABLE; OpenRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:46,683 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=178, state=RUNNABLE; OpenRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:46,685 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:46,685 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145646685"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145646685"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145646685"}]},"ts":"1676145646685"} 2023-02-11 20:00:46,686 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=181, state=RUNNABLE; OpenRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:46,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-11 20:00:46,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:46,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 20:00:46,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:46,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:46,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:46,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:46,839 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:46,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:46,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 20:00:46,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:46,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:46,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:46,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:46,841 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:46,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 20:00:46,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:46,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:46,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:46,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:46,842 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf 2023-02-11 20:00:46,842 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf 2023-02-11 20:00:46,842 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a4676c43c8662115dfd922e9ff72451 columnFamilyName cf 2023-02-11 20:00:46,847 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:46,847 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:46,847 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf 2023-02-11 20:00:46,848 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf 2023-02-11 20:00:46,848 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf 2023-02-11 20:00:46,848 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf 2023-02-11 20:00:46,848 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region db772e5c38666094ec2e3b8fb5e805c2 columnFamilyName cf 2023-02-11 20:00:46,849 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f1fc9bf98611ec0266597565431cc2a1 columnFamilyName cf 2023-02-11 20:00:46,864 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-0bcc1b49b65f42d389fcb662a909dbf1 2023-02-11 20:00:46,865 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-c116febc28fc423781baa2c67a3e1d2f 2023-02-11 20:00:46,865 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-85a30970077e4a559c5f7e7c79bd9336 2023-02-11 20:00:46,878 DEBUG [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-cda13ed20aa14162b1be58c678d7c72c 2023-02-11 20:00:46,878 INFO [StoreOpener-1a4676c43c8662115dfd922e9ff72451-1] regionserver.HStore(310): Store=1a4676c43c8662115dfd922e9ff72451/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:46,879 DEBUG [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-e793845eb1f447169fd725e5a836da1b 2023-02-11 20:00:46,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:46,880 INFO [StoreOpener-db772e5c38666094ec2e3b8fb5e805c2-1] regionserver.HStore(310): Store=db772e5c38666094ec2e3b8fb5e805c2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:46,880 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:46,880 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:46,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:46,883 DEBUG [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-f1723f83b8234b60b4690710a470498b 2023-02-11 20:00:46,883 INFO [StoreOpener-f1fc9bf98611ec0266597565431cc2a1-1] regionserver.HStore(310): Store=f1fc9bf98611ec0266597565431cc2a1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:46,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:46,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:46,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:46,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:46,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 20:00:46,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 20:00:46,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1a4676c43c8662115dfd922e9ff72451; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67459463, jitterRate=0.005224332213401794}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:46,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened db772e5c38666094ec2e3b8fb5e805c2; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75436959, jitterRate=0.1240982860326767}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:46,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1a4676c43c8662115dfd922e9ff72451: Running coprocessor pre-open hook at 1676145646838Writing region info on filesystem at 1676145646838Initializing all the Stores at 1676145646839 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145646839Cleaning up temporary data from old regions at 1676145646880 (+41 ms)Cleaning up detritus from prior splits at 1676145646881 (+1 ms)Running coprocessor post-open hooks at 1676145646889 (+8 ms)Region opened successfully at 1676145646889 2023-02-11 20:00:46,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:46,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for db772e5c38666094ec2e3b8fb5e805c2: Running coprocessor pre-open hook at 1676145646841Writing region info on filesystem at 1676145646841Initializing all the Stores at 1676145646841Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145646841Cleaning up temporary data from old regions at 1676145646882 (+41 ms)Cleaning up detritus from prior splits at 1676145646883 (+1 ms)Running coprocessor post-open hooks at 1676145646889 (+6 ms)Region opened successfully at 1676145646889 2023-02-11 20:00:46,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451., pid=187, masterSystemTime=1676145646833 2023-02-11 20:00:46,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 20:00:46,891 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f1fc9bf98611ec0266597565431cc2a1; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71567405, jitterRate=0.06643743813037872}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:46,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f1fc9bf98611ec0266597565431cc2a1: Running coprocessor pre-open hook at 1676145646841Writing region info on filesystem at 1676145646841Initializing all the Stores at 1676145646842 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145646842Cleaning up temporary data from old regions at 1676145646886 (+44 ms)Cleaning up detritus from prior splits at 1676145646886Running coprocessor post-open hooks at 1676145646891 (+5 ms)Region opened successfully at 1676145646891 2023-02-11 20:00:46,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:46,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:46,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:46,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 20:00:46,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1., pid=185, masterSystemTime=1676145646834 2023-02-11 20:00:46,892 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:46,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:46,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:46,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:46,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:46,892 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145646892"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145646892"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145646892"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145646892"}]},"ts":"1676145646892"} 2023-02-11 20:00:46,895 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:46,895 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:46,895 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:46,895 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:46,895 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 20:00:46,895 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:46,895 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:46,896 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145646895"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145646895"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145646895"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145646895"}]},"ts":"1676145646895"} 2023-02-11 20:00:46,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:46,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:46,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:46,896 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2., pid=182, masterSystemTime=1676145646833 2023-02-11 20:00:46,896 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf 2023-02-11 20:00:46,896 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf 2023-02-11 20:00:46,897 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c630b1709e678e0b00c070c7cac86d30 columnFamilyName cf 2023-02-11 20:00:46,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:46,898 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:46,898 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:46,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 20:00:46,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676145622835 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:46,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:46,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:46,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:46,899 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=181 2023-02-11 20:00:46,899 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:46,899 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=181, state=SUCCESS; OpenRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,35937,1676145619887 in 209 msec 2023-02-11 20:00:46,899 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145646899"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145646899"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145646899"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145646899"}]},"ts":"1676145646899"} 2023-02-11 20:00:46,901 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, ASSIGN in 383 msec 2023-02-11 20:00:46,901 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=177 2023-02-11 20:00:46,901 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=177, state=SUCCESS; OpenRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,44175,1676145619924 in 216 msec 2023-02-11 20:00:46,902 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, ASSIGN in 385 msec 2023-02-11 20:00:46,903 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=176 2023-02-11 20:00:46,903 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=176, state=SUCCESS; OpenRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841 in 226 msec 2023-02-11 20:00:46,904 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, ASSIGN in 387 msec 2023-02-11 20:00:46,904 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:46,905 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:46,905 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf 2023-02-11 20:00:46,905 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf 2023-02-11 20:00:46,906 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7c040ebb400ff6a8415e55ad85bbbe0e columnFamilyName cf 2023-02-11 20:00:46,906 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf 2023-02-11 20:00:46,908 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf 2023-02-11 20:00:46,908 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8de911ca888e28adfca434ff163cbba9 columnFamilyName cf 2023-02-11 20:00:46,913 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-5eb756b4ee3a4959a80c36b5b304212c 2023-02-11 20:00:46,918 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-233710642ffe449384153b4775e7e690 2023-02-11 20:00:46,919 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-d185ce9140d04e959efc0b58ace714af 2023-02-11 20:00:46,922 DEBUG [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-cf1d617c62bd452786cb06efe103dd00 2023-02-11 20:00:46,922 INFO [StoreOpener-c630b1709e678e0b00c070c7cac86d30-1] regionserver.HStore(310): Store=c630b1709e678e0b00c070c7cac86d30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:46,923 DEBUG [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-2e4307f987394952952cf7c59270b4e7 2023-02-11 20:00:46,923 INFO [StoreOpener-7c040ebb400ff6a8415e55ad85bbbe0e-1] regionserver.HStore(310): Store=7c040ebb400ff6a8415e55ad85bbbe0e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:46,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:46,923 DEBUG [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-e8a8517d5b3949c39858b0093dcf6481 2023-02-11 20:00:46,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:46,924 INFO [StoreOpener-8de911ca888e28adfca434ff163cbba9-1] regionserver.HStore(310): Store=8de911ca888e28adfca434ff163cbba9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:46,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:46,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:46,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:46,925 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:46,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:46,927 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:46,927 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:46,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 20:00:46,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 20:00:46,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 20:00:46,929 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c630b1709e678e0b00c070c7cac86d30; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72813836, jitterRate=0.08501070737838745}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:46,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c630b1709e678e0b00c070c7cac86d30: Running coprocessor pre-open hook at 1676145646893Writing region info on filesystem at 1676145646893Initializing all the Stores at 1676145646894 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145646895 (+1 ms)Cleaning up temporary data from old regions at 1676145646924 (+29 ms)Cleaning up detritus from prior splits at 1676145646924Running coprocessor post-open hooks at 1676145646929 (+5 ms)Region opened successfully at 1676145646929 2023-02-11 20:00:46,929 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7c040ebb400ff6a8415e55ad85bbbe0e; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64342556, jitterRate=-0.04122120141983032}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:46,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Running coprocessor pre-open hook at 1676145646896Writing region info on filesystem at 1676145646896Initializing all the Stores at 1676145646897 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145646897Cleaning up temporary data from old regions at 1676145646924 (+27 ms)Cleaning up detritus from prior splits at 1676145646925 (+1 ms)Running coprocessor post-open hooks at 1676145646929 (+4 ms)Region opened successfully at 1676145646929 2023-02-11 20:00:46,929 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 8de911ca888e28adfca434ff163cbba9; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67422338, jitterRate=0.0046711266040802}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:46,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 8de911ca888e28adfca434ff163cbba9: Running coprocessor pre-open hook at 1676145646898Writing region info on filesystem at 1676145646898Initializing all the Stores at 1676145646899 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145646899Cleaning up temporary data from old regions at 1676145646925 (+26 ms)Cleaning up detritus from prior splits at 1676145646925Running coprocessor post-open hooks at 1676145646929 (+4 ms)Region opened successfully at 1676145646929 2023-02-11 20:00:46,930 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30., pid=183, masterSystemTime=1676145646833 2023-02-11 20:00:46,930 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9., pid=184, masterSystemTime=1676145646833 2023-02-11 20:00:46,930 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e., pid=186, masterSystemTime=1676145646834 2023-02-11 20:00:46,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:46,932 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:46,932 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:46,932 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145646932"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145646932"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145646932"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145646932"}]},"ts":"1676145646932"} 2023-02-11 20:00:46,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:46,933 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:46,933 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:46,933 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145646933"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145646933"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145646933"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145646933"}]},"ts":"1676145646933"} 2023-02-11 20:00:46,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:46,933 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:46,939 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:46,939 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145646939"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145646939"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145646939"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145646939"}]},"ts":"1676145646939"} 2023-02-11 20:00:46,940 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=179 2023-02-11 20:00:46,940 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=178 2023-02-11 20:00:46,940 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=179, state=SUCCESS; OpenRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887 in 262 msec 2023-02-11 20:00:46,940 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=178, state=SUCCESS; OpenRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924 in 254 msec 2023-02-11 20:00:46,941 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, ASSIGN in 424 msec 2023-02-11 20:00:46,941 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, ASSIGN in 424 msec 2023-02-11 20:00:46,942 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=180 2023-02-11 20:00:46,942 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=180, state=SUCCESS; OpenRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841 in 259 msec 2023-02-11 20:00:46,943 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=180, resume processing ppid=175 2023-02-11 20:00:46,943 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, ASSIGN in 426 msec 2023-02-11 20:00:46,944 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145646944"}]},"ts":"1676145646944"} 2023-02-11 20:00:46,945 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=ENABLED in hbase:meta 2023-02-11 20:00:46,951 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1676145622835 snapshot=name: "snaptb1-1676145622835" table: "testRestoreSnapshot-1676145622835" creation_time: 1676145634760 type: DISABLED version: 2 ttl: 0 ) in 951 msec 2023-02-11 20:00:47,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-11 20:00:47,510 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676145622835, procId: 175 completed 2023-02-11 20:00:47,512 DEBUG [Listener at localhost/33691] zookeeper.ReadOnlyZKClient(139): Connect 0x69705b98 to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:47,540 DEBUG [Listener at localhost/33691] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e1138f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:47,544 DEBUG [hconnection-0x594888d9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:47,546 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53540, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:47,550 DEBUG [hconnection-0x594888d9-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:47,551 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52404, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:47,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,826 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 20:00:47,832 DEBUG [hconnection-0x594888d9-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:47,833 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58630, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:47,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:47,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,058 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 20:00:48,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,363 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 20:00:48,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,592 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 20:00:48,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,821 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 20:00:48,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:48,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,530 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70b95fe7 to 127.0.0.1:51732 2023-02-11 20:00:49,530 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:00:49,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:49,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:50,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,128 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 20:00:51,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,779 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676145622835' 2023-02-11 20:00:51,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:51,927 INFO [Listener at localhost/33691] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676145622835 2023-02-11 20:00:51,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676145622835 2023-02-11 20:00:51,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:51,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-11 20:00:51,932 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145651932"}]},"ts":"1676145651932"} 2023-02-11 20:00:51,934 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=DISABLING in hbase:meta 2023-02-11 20:00:51,936 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676145622835 to state=DISABLING 2023-02-11 20:00:51,937 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN}] 2023-02-11 20:00:51,938 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN 2023-02-11 20:00:51,940 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN 2023-02-11 20:00:51,940 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN 2023-02-11 20:00:51,941 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN 2023-02-11 20:00:51,941 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN 2023-02-11 20:00:51,941 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:51,941 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:51,941 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145651941"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145651941"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145651941"}]},"ts":"1676145651941"} 2023-02-11 20:00:51,941 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145651941"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145651941"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145651941"}]},"ts":"1676145651941"} 2023-02-11 20:00:51,942 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:51,942 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145651942"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145651942"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145651942"}]},"ts":"1676145651942"} 2023-02-11 20:00:51,942 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:51,942 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145651942"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145651942"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145651942"}]},"ts":"1676145651942"} 2023-02-11 20:00:51,943 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:51,943 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145651943"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145651943"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145651943"}]},"ts":"1676145651943"} 2023-02-11 20:00:51,944 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=194, state=RUNNABLE; CloseRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:51,945 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=192, state=RUNNABLE; CloseRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:51,946 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=191, state=RUNNABLE; CloseRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:51,951 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=190, state=RUNNABLE; CloseRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:51,951 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=189, state=RUNNABLE; CloseRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:51,952 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN 2023-02-11 20:00:51,954 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:51,954 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145651953"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145651953"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145651953"}]},"ts":"1676145651953"} 2023-02-11 20:00:51,956 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=193, state=RUNNABLE; CloseRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:52,098 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:52,099 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1a4676c43c8662115dfd922e9ff72451, disabling compactions & flushes 2023-02-11 20:00:52,099 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:52,099 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:52,099 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. after waiting 0 ms 2023-02-11 20:00:52,099 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:52,107 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:52,107 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:52,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7c040ebb400ff6a8415e55ad85bbbe0e, disabling compactions & flushes 2023-02-11 20:00:52,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:52,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:52,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. after waiting 0 ms 2023-02-11 20:00:52,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:52,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 8de911ca888e28adfca434ff163cbba9, disabling compactions & flushes 2023-02-11 20:00:52,109 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:52,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:52,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. after waiting 0 ms 2023-02-11 20:00:52,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:52,143 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 20:00:52,145 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451. 2023-02-11 20:00:52,146 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1a4676c43c8662115dfd922e9ff72451: Waiting for close lock at 1676145652098Running coprocessor pre-close hooks at 1676145652098Disabling compacts and flushes for region at 1676145652098Disabling writes for close at 1676145652099 (+1 ms)Writing region close event to WAL at 1676145652107 (+8 ms)Running coprocessor post-close hooks at 1676145652144 (+37 ms)Closed at 1676145652145 (+1 ms) 2023-02-11 20:00:52,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:52,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:52,164 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c630b1709e678e0b00c070c7cac86d30, disabling compactions & flushes 2023-02-11 20:00:52,164 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:52,164 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:52,164 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. after waiting 0 ms 2023-02-11 20:00:52,165 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:52,165 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=1a4676c43c8662115dfd922e9ff72451, regionState=CLOSED 2023-02-11 20:00:52,165 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145652165"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145652165"}]},"ts":"1676145652165"} 2023-02-11 20:00:52,170 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=194 2023-02-11 20:00:52,170 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=194, state=SUCCESS; CloseRegionProcedure 1a4676c43c8662115dfd922e9ff72451, server=jenkins-hbase4.apache.org,35937,1676145619887 in 224 msec 2023-02-11 20:00:52,172 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=1a4676c43c8662115dfd922e9ff72451, UNASSIGN in 233 msec 2023-02-11 20:00:52,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 20:00:52,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-11 20:00:52,200 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9. 2023-02-11 20:00:52,200 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 8de911ca888e28adfca434ff163cbba9: Waiting for close lock at 1676145652108Running coprocessor pre-close hooks at 1676145652108Disabling compacts and flushes for region at 1676145652108Disabling writes for close at 1676145652109 (+1 ms)Writing region close event to WAL at 1676145652144 (+35 ms)Running coprocessor post-close hooks at 1676145652173 (+29 ms)Closed at 1676145652200 (+27 ms) 2023-02-11 20:00:52,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 20:00:52,215 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:52,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:52,217 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=8de911ca888e28adfca434ff163cbba9, regionState=CLOSED 2023-02-11 20:00:52,217 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145652217"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145652217"}]},"ts":"1676145652217"} 2023-02-11 20:00:52,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e. 2023-02-11 20:00:52,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7c040ebb400ff6a8415e55ad85bbbe0e: Waiting for close lock at 1676145652107Running coprocessor pre-close hooks at 1676145652107Disabling compacts and flushes for region at 1676145652107Disabling writes for close at 1676145652108 (+1 ms)Writing region close event to WAL at 1676145652144 (+36 ms)Running coprocessor post-close hooks at 1676145652215 (+71 ms)Closed at 1676145652221 (+6 ms) 2023-02-11 20:00:52,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing db772e5c38666094ec2e3b8fb5e805c2, disabling compactions & flushes 2023-02-11 20:00:52,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:52,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:52,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. after waiting 0 ms 2023-02-11 20:00:52,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:52,233 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:52,233 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:52,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f1fc9bf98611ec0266597565431cc2a1, disabling compactions & flushes 2023-02-11 20:00:52,234 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:52,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:52,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. after waiting 0 ms 2023-02-11 20:00:52,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:52,237 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=7c040ebb400ff6a8415e55ad85bbbe0e, regionState=CLOSED 2023-02-11 20:00:52,237 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145652237"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145652237"}]},"ts":"1676145652237"} 2023-02-11 20:00:52,240 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=193 2023-02-11 20:00:52,240 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=193, state=SUCCESS; CloseRegionProcedure 8de911ca888e28adfca434ff163cbba9, server=jenkins-hbase4.apache.org,42011,1676145619841 in 272 msec 2023-02-11 20:00:52,244 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=8de911ca888e28adfca434ff163cbba9, UNASSIGN in 304 msec 2023-02-11 20:00:52,247 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=191 2023-02-11 20:00:52,247 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=191, state=SUCCESS; CloseRegionProcedure 7c040ebb400ff6a8415e55ad85bbbe0e, server=jenkins-hbase4.apache.org,44175,1676145619924 in 295 msec 2023-02-11 20:00:52,248 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=7c040ebb400ff6a8415e55ad85bbbe0e, UNASSIGN in 310 msec 2023-02-11 20:00:52,255 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 20:00:52,257 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30. 2023-02-11 20:00:52,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c630b1709e678e0b00c070c7cac86d30: Waiting for close lock at 1676145652149Running coprocessor pre-close hooks at 1676145652149Disabling compacts and flushes for region at 1676145652149Disabling writes for close at 1676145652165 (+16 ms)Writing region close event to WAL at 1676145652228 (+63 ms)Running coprocessor post-close hooks at 1676145652256 (+28 ms)Closed at 1676145652257 (+1 ms) 2023-02-11 20:00:52,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 20:00:52,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:52,261 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=c630b1709e678e0b00c070c7cac86d30, regionState=CLOSED 2023-02-11 20:00:52,261 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145652261"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145652261"}]},"ts":"1676145652261"} 2023-02-11 20:00:52,262 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2. 2023-02-11 20:00:52,262 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for db772e5c38666094ec2e3b8fb5e805c2: Waiting for close lock at 1676145652216Running coprocessor pre-close hooks at 1676145652216Disabling compacts and flushes for region at 1676145652216Disabling writes for close at 1676145652221 (+5 ms)Writing region close event to WAL at 1676145652232 (+11 ms)Running coprocessor post-close hooks at 1676145652259 (+27 ms)Closed at 1676145652262 (+3 ms) 2023-02-11 20:00:52,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:52,265 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=db772e5c38666094ec2e3b8fb5e805c2, regionState=CLOSED 2023-02-11 20:00:52,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 20:00:52,266 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676145652265"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145652265"}]},"ts":"1676145652265"} 2023-02-11 20:00:52,268 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=192 2023-02-11 20:00:52,268 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1. 2023-02-11 20:00:52,268 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=192, state=SUCCESS; CloseRegionProcedure c630b1709e678e0b00c070c7cac86d30, server=jenkins-hbase4.apache.org,35937,1676145619887 in 318 msec 2023-02-11 20:00:52,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f1fc9bf98611ec0266597565431cc2a1: Waiting for close lock at 1676145652233Running coprocessor pre-close hooks at 1676145652233Disabling compacts and flushes for region at 1676145652233Disabling writes for close at 1676145652234 (+1 ms)Writing region close event to WAL at 1676145652255 (+21 ms)Running coprocessor post-close hooks at 1676145652267 (+12 ms)Closed at 1676145652268 (+1 ms) 2023-02-11 20:00:52,272 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=c630b1709e678e0b00c070c7cac86d30, UNASSIGN in 331 msec 2023-02-11 20:00:52,273 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:52,273 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=f1fc9bf98611ec0266597565431cc2a1, regionState=CLOSED 2023-02-11 20:00:52,273 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676145652273"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145652273"}]},"ts":"1676145652273"} 2023-02-11 20:00:52,274 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=189 2023-02-11 20:00:52,274 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=189, state=SUCCESS; CloseRegionProcedure db772e5c38666094ec2e3b8fb5e805c2, server=jenkins-hbase4.apache.org,42011,1676145619841 in 321 msec 2023-02-11 20:00:52,275 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=db772e5c38666094ec2e3b8fb5e805c2, UNASSIGN in 337 msec 2023-02-11 20:00:52,276 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=190 2023-02-11 20:00:52,276 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=190, state=SUCCESS; CloseRegionProcedure f1fc9bf98611ec0266597565431cc2a1, server=jenkins-hbase4.apache.org,44175,1676145619924 in 328 msec 2023-02-11 20:00:52,278 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=190, resume processing ppid=188 2023-02-11 20:00:52,278 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676145622835, region=f1fc9bf98611ec0266597565431cc2a1, UNASSIGN in 339 msec 2023-02-11 20:00:52,278 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145652278"}]},"ts":"1676145652278"} 2023-02-11 20:00:52,281 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676145622835, state=DISABLED in hbase:meta 2023-02-11 20:00:52,284 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676145622835 to state=DISABLED 2023-02-11 20:00:52,288 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676145622835 in 356 msec 2023-02-11 20:00:52,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-11 20:00:52,685 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676145622835, procId: 188 completed 2023-02-11 20:00:52,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676145622835 2023-02-11 20:00:52,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:52,689 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:52,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-11 20:00:52,691 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:52,700 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:52,700 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:52,700 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:52,701 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:52,701 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:52,701 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:52,706 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/recovered.edits] 2023-02-11 20:00:52,706 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/recovered.edits] 2023-02-11 20:00:52,706 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/recovered.edits] 2023-02-11 20:00:52,707 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/recovered.edits] 2023-02-11 20:00:52,707 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/recovered.edits] 2023-02-11 20:00:52,708 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/recovered.edits] 2023-02-11 20:00:52,718 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-d185ce9140d04e959efc0b58ace714af already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-d185ce9140d04e959efc0b58ace714af. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645733 currentMtime: 1676145646102 2023-02-11 20:00:52,718 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-85a30970077e4a559c5f7e7c79bd9336 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-85a30970077e4a559c5f7e7c79bd9336. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645732 currentMtime: 1676145646501 2023-02-11 20:00:52,718 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-d185ce9140d04e959efc0b58ace714af to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-d185ce9140d04e959efc0b58ace714af.1676145652707. 2023-02-11 20:00:52,719 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-0bcc1b49b65f42d389fcb662a909dbf1 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-0bcc1b49b65f42d389fcb662a909dbf1. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645732 currentMtime: 1676145646101 2023-02-11 20:00:52,719 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-233710642ffe449384153b4775e7e690 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-233710642ffe449384153b4775e7e690. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645731 currentMtime: 1676145646101 2023-02-11 20:00:52,719 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-85a30970077e4a559c5f7e7c79bd9336 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-85a30970077e4a559c5f7e7c79bd9336.1676145652706. 2023-02-11 20:00:52,719 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-c116febc28fc423781baa2c67a3e1d2f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-c116febc28fc423781baa2c67a3e1d2f. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645732 currentMtime: 1676145646492 2023-02-11 20:00:52,719 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-233710642ffe449384153b4775e7e690 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-233710642ffe449384153b4775e7e690.1676145652707. 2023-02-11 20:00:52,719 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-0bcc1b49b65f42d389fcb662a909dbf1 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-0bcc1b49b65f42d389fcb662a909dbf1.1676145652708. 2023-02-11 20:00:52,719 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-5eb756b4ee3a4959a80c36b5b304212c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-5eb756b4ee3a4959a80c36b5b304212c. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645733 currentMtime: 1676145646099 2023-02-11 20:00:52,720 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-d185ce9140d04e959efc0b58ace714af to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-d185ce9140d04e959efc0b58ace714af 2023-02-11 20:00:52,720 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-85a30970077e4a559c5f7e7c79bd9336 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-85a30970077e4a559c5f7e7c79bd9336 2023-02-11 20:00:52,721 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-c116febc28fc423781baa2c67a3e1d2f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-c116febc28fc423781baa2c67a3e1d2f.1676145652707. 2023-02-11 20:00:52,721 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-5eb756b4ee3a4959a80c36b5b304212c to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-5eb756b4ee3a4959a80c36b5b304212c.1676145652706. 2023-02-11 20:00:52,721 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-233710642ffe449384153b4775e7e690 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-233710642ffe449384153b4775e7e690 2023-02-11 20:00:52,722 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-0bcc1b49b65f42d389fcb662a909dbf1 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-0bcc1b49b65f42d389fcb662a909dbf1 2023-02-11 20:00:52,723 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-c116febc28fc423781baa2c67a3e1d2f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-c116febc28fc423781baa2c67a3e1d2f 2023-02-11 20:00:52,723 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-5eb756b4ee3a4959a80c36b5b304212c to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-5eb756b4ee3a4959a80c36b5b304212c 2023-02-11 20:00:52,724 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-e793845eb1f447169fd725e5a836da1b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-e793845eb1f447169fd725e5a836da1b. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645734 currentMtime: 1676145646502 2023-02-11 20:00:52,725 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-e8a8517d5b3949c39858b0093dcf6481 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-e8a8517d5b3949c39858b0093dcf6481. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645735 currentMtime: 1676145646105 2023-02-11 20:00:52,725 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-e793845eb1f447169fd725e5a836da1b to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-e793845eb1f447169fd725e5a836da1b.1676145652706. 2023-02-11 20:00:52,726 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-e8a8517d5b3949c39858b0093dcf6481 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-e8a8517d5b3949c39858b0093dcf6481.1676145652707. 2023-02-11 20:00:52,726 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-cda13ed20aa14162b1be58c678d7c72c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-cda13ed20aa14162b1be58c678d7c72c. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645735 currentMtime: 1676145646105 2023-02-11 20:00:52,726 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-2e4307f987394952952cf7c59270b4e7 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-2e4307f987394952952cf7c59270b4e7. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645733 currentMtime: 1676145646105 2023-02-11 20:00:52,727 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-2e4307f987394952952cf7c59270b4e7 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-2e4307f987394952952cf7c59270b4e7.1676145652707. 2023-02-11 20:00:52,727 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-cda13ed20aa14162b1be58c678d7c72c to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-cda13ed20aa14162b1be58c678d7c72c.1676145652708. 2023-02-11 20:00:52,728 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-f1723f83b8234b60b4690710a470498b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-f1723f83b8234b60b4690710a470498b. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645735 currentMtime: 1676145646494 2023-02-11 20:00:52,728 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-e793845eb1f447169fd725e5a836da1b to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/cf/testRestoreSnapshot-1676145622835=db772e5c38666094ec2e3b8fb5e805c2-e793845eb1f447169fd725e5a836da1b 2023-02-11 20:00:52,729 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-e8a8517d5b3949c39858b0093dcf6481 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/cf/testRestoreSnapshot-1676145622835=8de911ca888e28adfca434ff163cbba9-e8a8517d5b3949c39858b0093dcf6481 2023-02-11 20:00:52,730 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-cf1d617c62bd452786cb06efe103dd00 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-cf1d617c62bd452786cb06efe103dd00. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645735 currentMtime: 1676145646102 2023-02-11 20:00:52,730 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-f1723f83b8234b60b4690710a470498b to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-f1723f83b8234b60b4690710a470498b.1676145652707. 2023-02-11 20:00:52,731 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-2e4307f987394952952cf7c59270b4e7 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/cf/testRestoreSnapshot-1676145622835=7c040ebb400ff6a8415e55ad85bbbe0e-2e4307f987394952952cf7c59270b4e7 2023-02-11 20:00:52,731 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-cda13ed20aa14162b1be58c678d7c72c to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/cf/testRestoreSnapshot-1676145622835=1a4676c43c8662115dfd922e9ff72451-cda13ed20aa14162b1be58c678d7c72c 2023-02-11 20:00:52,731 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-cf1d617c62bd452786cb06efe103dd00 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-cf1d617c62bd452786cb06efe103dd00.1676145652706. 2023-02-11 20:00:52,732 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-f1723f83b8234b60b4690710a470498b to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/cf/testRestoreSnapshot-1676145622835=f1fc9bf98611ec0266597565431cc2a1-f1723f83b8234b60b4690710a470498b 2023-02-11 20:00:52,733 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-cf1d617c62bd452786cb06efe103dd00 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/cf/testRestoreSnapshot-1676145622835=c630b1709e678e0b00c070c7cac86d30-cf1d617c62bd452786cb06efe103dd00 2023-02-11 20:00:52,737 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/recovered.edits/18.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9/recovered.edits/18.seqid 2023-02-11 20:00:52,738 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/8de911ca888e28adfca434ff163cbba9 2023-02-11 20:00:52,739 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/recovered.edits/18.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451/recovered.edits/18.seqid 2023-02-11 20:00:52,739 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/recovered.edits/18.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2/recovered.edits/18.seqid 2023-02-11 20:00:52,739 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/recovered.edits/18.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e/recovered.edits/18.seqid 2023-02-11 20:00:52,739 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/recovered.edits/18.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30/recovered.edits/18.seqid 2023-02-11 20:00:52,740 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/recovered.edits/18.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1/recovered.edits/18.seqid 2023-02-11 20:00:52,742 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/db772e5c38666094ec2e3b8fb5e805c2 2023-02-11 20:00:52,742 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/1a4676c43c8662115dfd922e9ff72451 2023-02-11 20:00:52,743 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/c630b1709e678e0b00c070c7cac86d30 2023-02-11 20:00:52,743 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/f1fc9bf98611ec0266597565431cc2a1 2023-02-11 20:00:52,744 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testRestoreSnapshot-1676145622835/7c040ebb400ff6a8415e55ad85bbbe0e 2023-02-11 20:00:52,745 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676145622835 regions 2023-02-11 20:00:52,746 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a 2023-02-11 20:00:52,746 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf] 2023-02-11 20:00:52,752 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645747 currentMtime: 1676145646026 2023-02-11 20:00:52,753 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868.1676145652746. 2023-02-11 20:00:52,754 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c202302112274319498d0488990bf52d9cf331868 2023-02-11 20:00:52,756 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645749 currentMtime: 1676145646028 2023-02-11 20:00:52,759 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de.1676145652746. 2023-02-11 20:00:52,762 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-a87ff679a2f3e71d9181a67b7542122c20230211711148e23ff84bf8ab386fd03d8bf1de 2023-02-11 20:00:52,764 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645749 currentMtime: 1676145646032 2023-02-11 20:00:52,765 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46.1676145652746. 2023-02-11 20:00:52,766 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021106d4a626bdfc4b2f8ab817113f23df46 2023-02-11 20:00:52,768 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645750 currentMtime: 1676145646036 2023-02-11 20:00:52,768 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27.1676145652746. 2023-02-11 20:00:52,769 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c4ca4238a0b923820dcc509a6f75849b2023021152181658b8854690b4ab85fb31ecef27 2023-02-11 20:00:52,770 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645752 currentMtime: 1676145646039 2023-02-11 20:00:52,771 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848.1676145652746. 2023-02-11 20:00:52,772 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302115c9eedbb9c4245cbbc882ae6ae502848 2023-02-11 20:00:52,774 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645753 currentMtime: 1676145646041 2023-02-11 20:00:52,775 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482.1676145652746. 2023-02-11 20:00:52,776 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-c81e728d9d4c2f636f067f89cc14862c202302117a7d641b6c1c451b9ed9423e5efb5482 2023-02-11 20:00:52,778 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645754 currentMtime: 1676145646043 2023-02-11 20:00:52,779 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071.1676145652746. 2023-02-11 20:00:52,781 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e202302112d2479608f89461d92bba08bfab8f071 2023-02-11 20:00:52,785 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645755 currentMtime: 1676145646046 2023-02-11 20:00:52,786 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d.1676145652746. 2023-02-11 20:00:52,786 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-d41d8cd98f00b204e9800998ecf8427e20230211ad91f886b6a44f019e4604b0c276970d 2023-02-11 20:00:52,787 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645756 currentMtime: 1676145646048 2023-02-11 20:00:52,788 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f.1676145652746. 2023-02-11 20:00:52,789 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d52023021176de1e2d4f494494b5eefd492b38fe4f 2023-02-11 20:00:52,790 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645757 currentMtime: 1676145646051 2023-02-11 20:00:52,790 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e.1676145652746. 2023-02-11 20:00:52,791 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-e4da3b7fbbce2345d7772b0674a318d5202302119b941bb9a9f14633b8720bb1ed1d320e 2023-02-11 20:00:52,792 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645758 currentMtime: 1676145646057 2023-02-11 20:00:52,793 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f.1676145652746. 2023-02-11 20:00:52,794 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf3202302113a902254056143b681ce96476366923f 2023-02-11 20:00:52,797 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47. archiveLen: 0 currentLen: 0 archiveMtime: 1676145645758 currentMtime: 1676145646059 2023-02-11 20:00:52,798 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47.1676145652746. 2023-02-11 20:00:52,799 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a/cf/testRestoreSnapshot-1676145622835=7cbf443d8efa7267c2d468f6e0b1360a-eccbc87e4b5ce2fe28308fd9f2a7baf320230211960ac9aa4cc746149e66ad3829bf9f47 2023-02-11 20:00:52,800 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testRestoreSnapshot-1676145622835/7cbf443d8efa7267c2d468f6e0b1360a 2023-02-11 20:00:52,802 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:52,806 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676145622835 from hbase:meta 2023-02-11 20:00:52,816 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676145622835' descriptor. 2023-02-11 20:00:52,818 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:52,818 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676145622835' from region states. 2023-02-11 20:00:52,818 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145652818"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:52,818 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145652818"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:52,818 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145652818"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:52,818 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145652818"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:52,818 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145652818"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:52,818 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145652818"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:52,826 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 20:00:52,826 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => db772e5c38666094ec2e3b8fb5e805c2, NAME => 'testRestoreSnapshot-1676145622835,,1676145622863.db772e5c38666094ec2e3b8fb5e805c2.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => f1fc9bf98611ec0266597565431cc2a1, NAME => 'testRestoreSnapshot-1676145622835,1,1676145622863.f1fc9bf98611ec0266597565431cc2a1.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 7c040ebb400ff6a8415e55ad85bbbe0e, NAME => 'testRestoreSnapshot-1676145622835,2,1676145622863.7c040ebb400ff6a8415e55ad85bbbe0e.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => c630b1709e678e0b00c070c7cac86d30, NAME => 'testRestoreSnapshot-1676145622835,3,1676145622863.c630b1709e678e0b00c070c7cac86d30.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 8de911ca888e28adfca434ff163cbba9, NAME => 'testRestoreSnapshot-1676145622835,4,1676145622863.8de911ca888e28adfca434ff163cbba9.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 1a4676c43c8662115dfd922e9ff72451, NAME => 'testRestoreSnapshot-1676145622835,5,1676145622863.1a4676c43c8662115dfd922e9ff72451.', STARTKEY => '5', ENDKEY => ''}] 2023-02-11 20:00:52,826 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676145622835' as deleted. 2023-02-11 20:00:52,826 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676145622835","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676145652826"}]},"ts":"9223372036854775807"} 2023-02-11 20:00:52,830 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676145622835 state from META 2023-02-11 20:00:52,833 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676145622835 2023-02-11 20:00:52,834 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676145622835 in 147 msec 2023-02-11 20:00:52,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-11 20:00:52,942 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676145622835, procId: 201 completed 2023-02-11 20:00:52,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676145622835" 2023-02-11 20:00:52,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676145622835 2023-02-11 20:00:52,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676145622835" 2023-02-11 20:00:52,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676145622835 2023-02-11 20:00:52,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb1-1676145622835" 2023-02-11 20:00:52,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1676145622835 2023-02-11 20:00:53,013 INFO [Listener at localhost/33691] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=407 (was 362) Potentially hanging thread: hconnection-0x6065ab53-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51732@0x69705b98-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6065ab53-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-68077062_15 at /127.0.0.1:36252 [Waiting for operation #507] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7415b278-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: hconnection-0x6065ab53-shared-pool-29 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51732@0x69705b98 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$45/262442535.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51732@0x69705b98-SendThread(127.0.0.1:51732) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-68077062_15 at /127.0.0.1:40072 [Waiting for operation #496] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1210470265_15 at /127.0.0.1:36266 [Waiting for operation #541] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x594888d9-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6065ab53-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-68077062_15 at /127.0.0.1:40076 [Waiting for operation #523] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-68077062_15 at /127.0.0.1:60392 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=794 (was 695) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=556 (was 399) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=3346 (was 4412) 2023-02-11 20:00:53,038 INFO [Listener at localhost/33691] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=407, OpenFileDescriptor=794, MaxFileDescriptor=60000, SystemLoadAverage=556, ProcessCount=168, AvailableMemoryMB=3344 2023-02-11 20:00:53,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'testCorruptedSnapshot-1676145653041', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-11 20:00:53,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1676145653041 2023-02-11 20:00:53,048 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676145653041 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 20:00:53,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1676145653041" procId is: 202 2023-02-11 20:00:53,049 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676145653041 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 20:00:53,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-11 20:00:53,058 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:53,058 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:53,059 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:53,059 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:53,059 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:53,059 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6 empty. 2023-02-11 20:00:53,059 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c empty. 2023-02-11 20:00:53,058 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:53,060 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:53,060 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183 empty. 2023-02-11 20:00:53,060 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:53,060 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e empty. 2023-02-11 20:00:53,061 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea empty. 2023-02-11 20:00:53,061 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:53,061 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c empty. 2023-02-11 20:00:53,061 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:53,061 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:53,061 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:53,061 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676145653041 regions 2023-02-11 20:00:53,096 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/.tabledesc/.tableinfo.0000000001 2023-02-11 20:00:53,098 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(7671): creating {ENCODED => 5bc4c474972588bf6c06dc75b6e4aea6, NAME => 'testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1676145653041', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:53,103 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(7671): creating {ENCODED => 7c1f214213ec2e6ba08d051362b2ca4e, NAME => 'testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1676145653041', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:53,104 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(7671): creating {ENCODED => 3518e490e7f4ccb7c60b36f15d4f9183, NAME => 'testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1676145653041', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:53,152 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:53,152 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(1603): Closing 5bc4c474972588bf6c06dc75b6e4aea6, disabling compactions & flushes 2023-02-11 20:00:53,152 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:53,153 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:53,153 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. after waiting 0 ms 2023-02-11 20:00:53,153 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:53,153 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:53,153 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(1557): Region close journal for 5bc4c474972588bf6c06dc75b6e4aea6: Waiting for close lock at 1676145653152Disabling compacts and flushes for region at 1676145653152Disabling writes for close at 1676145653153 (+1 ms)Writing region close event to WAL at 1676145653153Closed at 1676145653153 2023-02-11 20:00:53,153 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(7671): creating {ENCODED => 2f7f00f697d9b55aa03c57a3213d5a7c, NAME => 'testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1676145653041', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:53,183 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:53,184 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(1603): Closing 3518e490e7f4ccb7c60b36f15d4f9183, disabling compactions & flushes 2023-02-11 20:00:53,184 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:53,184 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:53,184 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. after waiting 0 ms 2023-02-11 20:00:53,184 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:53,184 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:53,184 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(1557): Region close journal for 3518e490e7f4ccb7c60b36f15d4f9183: Waiting for close lock at 1676145653184Disabling compacts and flushes for region at 1676145653184Disabling writes for close at 1676145653184Writing region close event to WAL at 1676145653184Closed at 1676145653184 2023-02-11 20:00:53,185 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(7671): creating {ENCODED => 38dcb256366159437c4db1abc8d1671c, NAME => 'testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1676145653041', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:53,194 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:53,194 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(1603): Closing 7c1f214213ec2e6ba08d051362b2ca4e, disabling compactions & flushes 2023-02-11 20:00:53,194 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:53,194 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:53,195 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. after waiting 0 ms 2023-02-11 20:00:53,195 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:53,195 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:53,195 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(1557): Region close journal for 7c1f214213ec2e6ba08d051362b2ca4e: Waiting for close lock at 1676145653194Disabling compacts and flushes for region at 1676145653194Disabling writes for close at 1676145653195 (+1 ms)Writing region close event to WAL at 1676145653195Closed at 1676145653195 2023-02-11 20:00:53,195 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(7671): creating {ENCODED => 3eae0a3c58bb2f53c1289450916a28ea, NAME => 'testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1676145653041', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp 2023-02-11 20:00:53,216 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:53,216 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(1603): Closing 2f7f00f697d9b55aa03c57a3213d5a7c, disabling compactions & flushes 2023-02-11 20:00:53,216 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:53,216 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:53,216 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. after waiting 0 ms 2023-02-11 20:00:53,216 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:53,216 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:53,217 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-0] regionserver.HRegion(1557): Region close journal for 2f7f00f697d9b55aa03c57a3213d5a7c: Waiting for close lock at 1676145653216Disabling compacts and flushes for region at 1676145653216Disabling writes for close at 1676145653216Writing region close event to WAL at 1676145653216Closed at 1676145653216 2023-02-11 20:00:53,233 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:53,233 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(1603): Closing 38dcb256366159437c4db1abc8d1671c, disabling compactions & flushes 2023-02-11 20:00:53,233 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:53,233 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:53,233 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. after waiting 0 ms 2023-02-11 20:00:53,233 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:53,233 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:53,233 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-2] regionserver.HRegion(1557): Region close journal for 38dcb256366159437c4db1abc8d1671c: Waiting for close lock at 1676145653233Disabling compacts and flushes for region at 1676145653233Disabling writes for close at 1676145653233Writing region close event to WAL at 1676145653233Closed at 1676145653233 2023-02-11 20:00:53,287 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:53,287 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(1603): Closing 3eae0a3c58bb2f53c1289450916a28ea, disabling compactions & flushes 2023-02-11 20:00:53,288 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:53,288 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:53,288 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. after waiting 0 ms 2023-02-11 20:00:53,288 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:53,288 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:53,288 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676145653041-pool-1] regionserver.HRegion(1557): Region close journal for 3eae0a3c58bb2f53c1289450916a28ea: Waiting for close lock at 1676145653287Disabling compacts and flushes for region at 1676145653287Disabling writes for close at 1676145653288 (+1 ms)Writing region close event to WAL at 1676145653288Closed at 1676145653288 2023-02-11 20:00:53,292 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676145653041 execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 20:00:53,294 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145653294"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145653294"}]},"ts":"1676145653294"} 2023-02-11 20:00:53,294 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653294"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145653294"}]},"ts":"1676145653294"} 2023-02-11 20:00:53,294 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653294"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145653294"}]},"ts":"1676145653294"} 2023-02-11 20:00:53,294 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653294"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145653294"}]},"ts":"1676145653294"} 2023-02-11 20:00:53,294 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653294"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145653294"}]},"ts":"1676145653294"} 2023-02-11 20:00:53,295 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145653294"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145653294"}]},"ts":"1676145653294"} 2023-02-11 20:00:53,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-11 20:00:53,308 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 20:00:53,309 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676145653041 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 20:00:53,309 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145653309"}]},"ts":"1676145653309"} 2023-02-11 20:00:53,310 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676145653041, state=ENABLING in hbase:meta 2023-02-11 20:00:53,318 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 20:00:53,321 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 20:00:53,321 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 20:00:53,321 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 20:00:53,321 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 20:00:53,322 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, ASSIGN}] 2023-02-11 20:00:53,325 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, ASSIGN 2023-02-11 20:00:53,325 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, ASSIGN 2023-02-11 20:00:53,325 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, ASSIGN 2023-02-11 20:00:53,326 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, ASSIGN 2023-02-11 20:00:53,326 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42011,1676145619841; forceNewPlan=false, retain=false 2023-02-11 20:00:53,327 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35937,1676145619887; forceNewPlan=false, retain=false 2023-02-11 20:00:53,327 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44175,1676145619924; forceNewPlan=false, retain=false 2023-02-11 20:00:53,327 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35937,1676145619887; forceNewPlan=false, retain=false 2023-02-11 20:00:53,327 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, ASSIGN 2023-02-11 20:00:53,328 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, ASSIGN 2023-02-11 20:00:53,328 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44175,1676145619924; forceNewPlan=false, retain=false 2023-02-11 20:00:53,329 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42011,1676145619841; forceNewPlan=false, retain=false 2023-02-11 20:00:53,477 INFO [jenkins-hbase4:39971] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 20:00:53,481 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=38dcb256366159437c4db1abc8d1671c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:53,481 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653481"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145653481"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145653481"}]},"ts":"1676145653481"} 2023-02-11 20:00:53,482 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=3518e490e7f4ccb7c60b36f15d4f9183, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:53,482 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653482"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145653482"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145653482"}]},"ts":"1676145653482"} 2023-02-11 20:00:53,482 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=5bc4c474972588bf6c06dc75b6e4aea6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:53,482 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=2f7f00f697d9b55aa03c57a3213d5a7c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:53,482 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653482"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145653482"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145653482"}]},"ts":"1676145653482"} 2023-02-11 20:00:53,483 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=7c1f214213ec2e6ba08d051362b2ca4e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:53,483 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653483"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145653483"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145653483"}]},"ts":"1676145653483"} 2023-02-11 20:00:53,482 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145653482"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145653482"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145653482"}]},"ts":"1676145653482"} 2023-02-11 20:00:53,489 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=207, state=RUNNABLE; OpenRegionProcedure 38dcb256366159437c4db1abc8d1671c, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:53,489 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=205, state=RUNNABLE; OpenRegionProcedure 3518e490e7f4ccb7c60b36f15d4f9183, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:53,490 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=206, state=RUNNABLE; OpenRegionProcedure 2f7f00f697d9b55aa03c57a3213d5a7c, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:53,491 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=204, state=RUNNABLE; OpenRegionProcedure 7c1f214213ec2e6ba08d051362b2ca4e, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:53,492 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=203, state=RUNNABLE; OpenRegionProcedure 5bc4c474972588bf6c06dc75b6e4aea6, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:53,497 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=3eae0a3c58bb2f53c1289450916a28ea, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:53,497 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145653497"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145653497"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145653497"}]},"ts":"1676145653497"} 2023-02-11 20:00:53,499 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=208, state=RUNNABLE; OpenRegionProcedure 3eae0a3c58bb2f53c1289450916a28ea, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:53,646 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:53,646 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5bc4c474972588bf6c06dc75b6e4aea6, NAME => 'testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 20:00:53,646 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:53,646 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:53,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:53,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:53,647 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:53,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7c1f214213ec2e6ba08d051362b2ca4e, NAME => 'testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 20:00:53,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:53,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:53,648 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:53,648 INFO [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:53,648 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:53,649 INFO [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:53,649 DEBUG [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf 2023-02-11 20:00:53,649 DEBUG [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf 2023-02-11 20:00:53,650 INFO [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5bc4c474972588bf6c06dc75b6e4aea6 columnFamilyName cf 2023-02-11 20:00:53,659 INFO [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] regionserver.HStore(310): Store=5bc4c474972588bf6c06dc75b6e4aea6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:53,664 DEBUG [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf 2023-02-11 20:00:53,664 DEBUG [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf 2023-02-11 20:00:53,664 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:53,664 INFO [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7c1f214213ec2e6ba08d051362b2ca4e columnFamilyName cf 2023-02-11 20:00:53,665 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:53,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 38dcb256366159437c4db1abc8d1671c, NAME => 'testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 20:00:53,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:53,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:53,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:53,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:53,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:53,666 INFO [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] regionserver.HStore(310): Store=7c1f214213ec2e6ba08d051362b2ca4e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:53,668 INFO [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:53,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:53,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:53,669 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:53,671 DEBUG [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf 2023-02-11 20:00:53,671 DEBUG [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf 2023-02-11 20:00:53,671 INFO [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 38dcb256366159437c4db1abc8d1671c columnFamilyName cf 2023-02-11 20:00:53,673 INFO [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] regionserver.HStore(310): Store=38dcb256366159437c4db1abc8d1671c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:53,673 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:53,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 20:00:53,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:53,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:53,675 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5bc4c474972588bf6c06dc75b6e4aea6; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63481439, jitterRate=-0.054052844643592834}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:53,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5bc4c474972588bf6c06dc75b6e4aea6: Running coprocessor pre-open hook at 1676145653647Writing region info on filesystem at 1676145653647Initializing all the Stores at 1676145653647Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145653647Cleaning up temporary data from old regions at 1676145653666 (+19 ms)Cleaning up detritus from prior splits at 1676145653666Running coprocessor post-open hooks at 1676145653675 (+9 ms)Region opened successfully at 1676145653675 2023-02-11 20:00:53,676 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6., pid=213, masterSystemTime=1676145653643 2023-02-11 20:00:53,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:53,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:53,679 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:53,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2f7f00f697d9b55aa03c57a3213d5a7c, NAME => 'testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 20:00:53,679 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=5bc4c474972588bf6c06dc75b6e4aea6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:53,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:53,679 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145653679"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145653679"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145653679"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145653679"}]},"ts":"1676145653679"} 2023-02-11 20:00:53,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:53,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:53,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:53,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:53,684 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=203 2023-02-11 20:00:53,684 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=203, state=SUCCESS; OpenRegionProcedure 5bc4c474972588bf6c06dc75b6e4aea6, server=jenkins-hbase4.apache.org,42011,1676145619841 in 189 msec 2023-02-11 20:00:53,686 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, ASSIGN in 363 msec 2023-02-11 20:00:53,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 20:00:53,688 INFO [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:53,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 20:00:53,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7c1f214213ec2e6ba08d051362b2ca4e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61377460, jitterRate=-0.08540457487106323}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:53,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7c1f214213ec2e6ba08d051362b2ca4e: Running coprocessor pre-open hook at 1676145653648Writing region info on filesystem at 1676145653648Initializing all the Stores at 1676145653648Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145653649 (+1 ms)Cleaning up temporary data from old regions at 1676145653670 (+21 ms)Cleaning up detritus from prior splits at 1676145653671 (+1 ms)Running coprocessor post-open hooks at 1676145653689 (+18 ms)Region opened successfully at 1676145653689 2023-02-11 20:00:53,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 38dcb256366159437c4db1abc8d1671c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69580235, jitterRate=0.036826297640800476}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:53,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e., pid=212, masterSystemTime=1676145653644 2023-02-11 20:00:53,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 38dcb256366159437c4db1abc8d1671c: Running coprocessor pre-open hook at 1676145653665Writing region info on filesystem at 1676145653665Initializing all the Stores at 1676145653666 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145653666Cleaning up temporary data from old regions at 1676145653676 (+10 ms)Cleaning up detritus from prior splits at 1676145653676Running coprocessor post-open hooks at 1676145653689 (+13 ms)Region opened successfully at 1676145653689 2023-02-11 20:00:53,690 DEBUG [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf 2023-02-11 20:00:53,690 DEBUG [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf 2023-02-11 20:00:53,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c., pid=209, masterSystemTime=1676145653643 2023-02-11 20:00:53,690 INFO [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2f7f00f697d9b55aa03c57a3213d5a7c columnFamilyName cf 2023-02-11 20:00:53,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:53,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:53,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:53,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3eae0a3c58bb2f53c1289450916a28ea, NAME => 'testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 20:00:53,691 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=7c1f214213ec2e6ba08d051362b2ca4e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:53,692 INFO [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] regionserver.HStore(310): Store=2f7f00f697d9b55aa03c57a3213d5a7c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:53,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:53,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:53,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:53,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:53,692 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653691"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145653691"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145653691"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145653691"}]},"ts":"1676145653691"} 2023-02-11 20:00:53,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:53,692 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:53,692 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:53,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3518e490e7f4ccb7c60b36f15d4f9183, NAME => 'testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 20:00:53,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:53,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:53,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:53,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:53,693 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=38dcb256366159437c4db1abc8d1671c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:53,693 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653693"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145653693"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145653693"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145653693"}]},"ts":"1676145653693"} 2023-02-11 20:00:53,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:53,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:53,695 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=204 2023-02-11 20:00:53,695 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=204, state=SUCCESS; OpenRegionProcedure 7c1f214213ec2e6ba08d051362b2ca4e, server=jenkins-hbase4.apache.org,35937,1676145619887 in 203 msec 2023-02-11 20:00:53,696 INFO [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:53,700 INFO [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:53,701 DEBUG [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf 2023-02-11 20:00:53,701 DEBUG [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf 2023-02-11 20:00:53,702 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, ASSIGN in 375 msec 2023-02-11 20:00:53,703 DEBUG [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf 2023-02-11 20:00:53,703 DEBUG [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf 2023-02-11 20:00:53,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:53,703 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=207 2023-02-11 20:00:53,703 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=207, state=SUCCESS; OpenRegionProcedure 38dcb256366159437c4db1abc8d1671c, server=jenkins-hbase4.apache.org,44175,1676145619924 in 207 msec 2023-02-11 20:00:53,703 INFO [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3eae0a3c58bb2f53c1289450916a28ea columnFamilyName cf 2023-02-11 20:00:53,704 INFO [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3518e490e7f4ccb7c60b36f15d4f9183 columnFamilyName cf 2023-02-11 20:00:53,704 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, ASSIGN in 382 msec 2023-02-11 20:00:53,705 INFO [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] regionserver.HStore(310): Store=3eae0a3c58bb2f53c1289450916a28ea/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:53,706 INFO [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] regionserver.HStore(310): Store=3518e490e7f4ccb7c60b36f15d4f9183/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:53,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:53,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:53,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:53,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 20:00:53,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:53,708 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2f7f00f697d9b55aa03c57a3213d5a7c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64973880, jitterRate=-0.031813740730285645}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:53,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2f7f00f697d9b55aa03c57a3213d5a7c: Running coprocessor pre-open hook at 1676145653679Writing region info on filesystem at 1676145653679Initializing all the Stores at 1676145653681 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145653681Cleaning up temporary data from old regions at 1676145653694 (+13 ms)Cleaning up detritus from prior splits at 1676145653695 (+1 ms)Running coprocessor post-open hooks at 1676145653708 (+13 ms)Region opened successfully at 1676145653708 2023-02-11 20:00:53,709 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c., pid=211, masterSystemTime=1676145653643 2023-02-11 20:00:53,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:53,711 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:53,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:53,711 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=2f7f00f697d9b55aa03c57a3213d5a7c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:53,712 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653711"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145653711"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145653711"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145653711"}]},"ts":"1676145653711"} 2023-02-11 20:00:53,715 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=206 2023-02-11 20:00:53,715 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:53,715 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=206, state=SUCCESS; OpenRegionProcedure 2f7f00f697d9b55aa03c57a3213d5a7c, server=jenkins-hbase4.apache.org,42011,1676145619841 in 224 msec 2023-02-11 20:00:53,717 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, ASSIGN in 394 msec 2023-02-11 20:00:53,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 20:00:53,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 20:00:53,739 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3518e490e7f4ccb7c60b36f15d4f9183; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64566234, jitterRate=-0.037888139486312866}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:53,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3518e490e7f4ccb7c60b36f15d4f9183: Running coprocessor pre-open hook at 1676145653693Writing region info on filesystem at 1676145653693Initializing all the Stores at 1676145653694 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145653694Cleaning up temporary data from old regions at 1676145653708 (+14 ms)Cleaning up detritus from prior splits at 1676145653708Running coprocessor post-open hooks at 1676145653739 (+31 ms)Region opened successfully at 1676145653739 2023-02-11 20:00:53,739 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3eae0a3c58bb2f53c1289450916a28ea; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65303205, jitterRate=-0.026906415820121765}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:53,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3eae0a3c58bb2f53c1289450916a28ea: Running coprocessor pre-open hook at 1676145653692Writing region info on filesystem at 1676145653692Initializing all the Stores at 1676145653693 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145653693Cleaning up temporary data from old regions at 1676145653708 (+15 ms)Cleaning up detritus from prior splits at 1676145653710 (+2 ms)Running coprocessor post-open hooks at 1676145653739 (+29 ms)Region opened successfully at 1676145653739 2023-02-11 20:00:53,739 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183., pid=210, masterSystemTime=1676145653643 2023-02-11 20:00:53,740 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea., pid=214, masterSystemTime=1676145653644 2023-02-11 20:00:53,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:53,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:53,741 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=3518e490e7f4ccb7c60b36f15d4f9183, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:53,742 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653741"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145653741"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145653741"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145653741"}]},"ts":"1676145653741"} 2023-02-11 20:00:53,742 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=3eae0a3c58bb2f53c1289450916a28ea, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:53,742 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145653742"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145653742"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145653742"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145653742"}]},"ts":"1676145653742"} 2023-02-11 20:00:53,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:53,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:53,753 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=205 2023-02-11 20:00:53,753 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=205, state=SUCCESS; OpenRegionProcedure 3518e490e7f4ccb7c60b36f15d4f9183, server=jenkins-hbase4.apache.org,44175,1676145619924 in 254 msec 2023-02-11 20:00:53,754 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=208 2023-02-11 20:00:53,754 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=208, state=SUCCESS; OpenRegionProcedure 3eae0a3c58bb2f53c1289450916a28ea, server=jenkins-hbase4.apache.org,35937,1676145619887 in 245 msec 2023-02-11 20:00:53,755 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, ASSIGN in 432 msec 2023-02-11 20:00:53,756 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=208, resume processing ppid=202 2023-02-11 20:00:53,757 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, ASSIGN in 433 msec 2023-02-11 20:00:53,759 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676145653041 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 20:00:53,759 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145653759"}]},"ts":"1676145653759"} 2023-02-11 20:00:53,762 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676145653041, state=ENABLED in hbase:meta 2023-02-11 20:00:53,769 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676145653041 execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 20:00:53,771 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1676145653041 in 725 msec 2023-02-11 20:00:53,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-11 20:00:53,806 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1676145653041, procId: 202 completed 2023-02-11 20:00:53,812 DEBUG [Listener at localhost/33691] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676145653041 2023-02-11 20:00:53,812 DEBUG [Listener at localhost/33691] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:53,816 INFO [Listener at localhost/33691] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 20:00:53,841 INFO [Listener at localhost/33691] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676145653041 2023-02-11 20:00:53,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676145653041 2023-02-11 20:00:53,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676145653041 2023-02-11 20:00:53,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-11 20:00:53,847 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145653847"}]},"ts":"1676145653847"} 2023-02-11 20:00:53,849 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676145653041, state=DISABLING in hbase:meta 2023-02-11 20:00:53,851 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676145653041 to state=DISABLING 2023-02-11 20:00:53,851 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, UNASSIGN}] 2023-02-11 20:00:53,856 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, UNASSIGN 2023-02-11 20:00:53,856 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, UNASSIGN 2023-02-11 20:00:53,856 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, UNASSIGN 2023-02-11 20:00:53,856 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, UNASSIGN 2023-02-11 20:00:53,856 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=5bc4c474972588bf6c06dc75b6e4aea6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:53,857 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, UNASSIGN 2023-02-11 20:00:53,857 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145653856"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145653856"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145653856"}]},"ts":"1676145653856"} 2023-02-11 20:00:53,857 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=3518e490e7f4ccb7c60b36f15d4f9183, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:53,857 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=7c1f214213ec2e6ba08d051362b2ca4e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:53,857 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653857"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145653857"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145653857"}]},"ts":"1676145653857"} 2023-02-11 20:00:53,857 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653857"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145653857"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145653857"}]},"ts":"1676145653857"} 2023-02-11 20:00:53,858 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=3eae0a3c58bb2f53c1289450916a28ea, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:53,858 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145653858"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145653858"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145653858"}]},"ts":"1676145653858"} 2023-02-11 20:00:53,858 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=2f7f00f697d9b55aa03c57a3213d5a7c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:53,858 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653857"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145653857"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145653857"}]},"ts":"1676145653857"} 2023-02-11 20:00:53,860 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=216, state=RUNNABLE; CloseRegionProcedure 5bc4c474972588bf6c06dc75b6e4aea6, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:53,861 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=218, state=RUNNABLE; CloseRegionProcedure 3518e490e7f4ccb7c60b36f15d4f9183, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:53,863 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=217, state=RUNNABLE; CloseRegionProcedure 7c1f214213ec2e6ba08d051362b2ca4e, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:53,864 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=221, state=RUNNABLE; CloseRegionProcedure 3eae0a3c58bb2f53c1289450916a28ea, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:53,864 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=219, state=RUNNABLE; CloseRegionProcedure 2f7f00f697d9b55aa03c57a3213d5a7c, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:53,867 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, UNASSIGN 2023-02-11 20:00:53,868 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=38dcb256366159437c4db1abc8d1671c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:53,868 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145653868"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145653868"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145653868"}]},"ts":"1676145653868"} 2023-02-11 20:00:53,872 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=220, state=RUNNABLE; CloseRegionProcedure 38dcb256366159437c4db1abc8d1671c, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:54,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:54,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2f7f00f697d9b55aa03c57a3213d5a7c, disabling compactions & flushes 2023-02-11 20:00:54,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:54,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:54,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. after waiting 0 ms 2023-02-11 20:00:54,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:54,018 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:54,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 38dcb256366159437c4db1abc8d1671c, disabling compactions & flushes 2023-02-11 20:00:54,018 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:54,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:54,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. after waiting 0 ms 2023-02-11 20:00:54,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:54,018 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:54,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3eae0a3c58bb2f53c1289450916a28ea, disabling compactions & flushes 2023-02-11 20:00:54,018 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:54,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:54,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. after waiting 0 ms 2023-02-11 20:00:54,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:54,022 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 20:00:54,023 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:54,023 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2f7f00f697d9b55aa03c57a3213d5a7c: Waiting for close lock at 1676145654015Running coprocessor pre-close hooks at 1676145654015Disabling compacts and flushes for region at 1676145654015Disabling writes for close at 1676145654015Writing region close event to WAL at 1676145654016 (+1 ms)Running coprocessor post-close hooks at 1676145654023 (+7 ms)Closed at 1676145654023 2023-02-11 20:00:54,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:54,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:54,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5bc4c474972588bf6c06dc75b6e4aea6, disabling compactions & flushes 2023-02-11 20:00:54,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:54,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:54,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. after waiting 0 ms 2023-02-11 20:00:54,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:54,026 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=2f7f00f697d9b55aa03c57a3213d5a7c, regionState=CLOSED 2023-02-11 20:00:54,026 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145654026"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145654026"}]},"ts":"1676145654026"} 2023-02-11 20:00:54,031 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=219 2023-02-11 20:00:54,031 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=219, state=SUCCESS; CloseRegionProcedure 2f7f00f697d9b55aa03c57a3213d5a7c, server=jenkins-hbase4.apache.org,42011,1676145619841 in 164 msec 2023-02-11 20:00:54,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 20:00:54,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 20:00:54,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:54,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:54,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5bc4c474972588bf6c06dc75b6e4aea6: Waiting for close lock at 1676145654025Running coprocessor pre-close hooks at 1676145654025Disabling compacts and flushes for region at 1676145654025Disabling writes for close at 1676145654025Writing region close event to WAL at 1676145654026 (+1 ms)Running coprocessor post-close hooks at 1676145654033 (+7 ms)Closed at 1676145654033 2023-02-11 20:00:54,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3eae0a3c58bb2f53c1289450916a28ea: Waiting for close lock at 1676145654018Running coprocessor pre-close hooks at 1676145654018Disabling compacts and flushes for region at 1676145654018Disabling writes for close at 1676145654018Writing region close event to WAL at 1676145654022 (+4 ms)Running coprocessor post-close hooks at 1676145654033 (+11 ms)Closed at 1676145654033 2023-02-11 20:00:54,040 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, UNASSIGN in 180 msec 2023-02-11 20:00:54,041 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:54,041 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 20:00:54,042 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=5bc4c474972588bf6c06dc75b6e4aea6, regionState=CLOSED 2023-02-11 20:00:54,043 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145654042"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145654042"}]},"ts":"1676145654042"} 2023-02-11 20:00:54,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:54,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:54,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7c1f214213ec2e6ba08d051362b2ca4e, disabling compactions & flushes 2023-02-11 20:00:54,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:54,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:54,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. after waiting 0 ms 2023-02-11 20:00:54,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:54,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:54,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 38dcb256366159437c4db1abc8d1671c: Waiting for close lock at 1676145654018Running coprocessor pre-close hooks at 1676145654018Disabling compacts and flushes for region at 1676145654018Disabling writes for close at 1676145654018Writing region close event to WAL at 1676145654032 (+14 ms)Running coprocessor post-close hooks at 1676145654043 (+11 ms)Closed at 1676145654043 2023-02-11 20:00:54,044 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=3eae0a3c58bb2f53c1289450916a28ea, regionState=CLOSED 2023-02-11 20:00:54,044 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145654044"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145654044"}]},"ts":"1676145654044"} 2023-02-11 20:00:54,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:54,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:54,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3518e490e7f4ccb7c60b36f15d4f9183, disabling compactions & flushes 2023-02-11 20:00:54,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:54,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:54,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. after waiting 0 ms 2023-02-11 20:00:54,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:54,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 20:00:54,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 20:00:54,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:54,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7c1f214213ec2e6ba08d051362b2ca4e: Waiting for close lock at 1676145654043Running coprocessor pre-close hooks at 1676145654043Disabling compacts and flushes for region at 1676145654043Disabling writes for close at 1676145654043Writing region close event to WAL at 1676145654045 (+2 ms)Running coprocessor post-close hooks at 1676145654052 (+7 ms)Closed at 1676145654052 2023-02-11 20:00:54,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:54,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3518e490e7f4ccb7c60b36f15d4f9183: Waiting for close lock at 1676145654047Running coprocessor pre-close hooks at 1676145654047Disabling compacts and flushes for region at 1676145654047Disabling writes for close at 1676145654047Writing region close event to WAL at 1676145654047Running coprocessor post-close hooks at 1676145654052 (+5 ms)Closed at 1676145654052 2023-02-11 20:00:54,064 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=38dcb256366159437c4db1abc8d1671c, regionState=CLOSED 2023-02-11 20:00:54,064 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145654064"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145654064"}]},"ts":"1676145654064"} 2023-02-11 20:00:54,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:54,066 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=216 2023-02-11 20:00:54,066 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=221 2023-02-11 20:00:54,066 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=216, state=SUCCESS; CloseRegionProcedure 5bc4c474972588bf6c06dc75b6e4aea6, server=jenkins-hbase4.apache.org,42011,1676145619841 in 185 msec 2023-02-11 20:00:54,066 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=221, state=SUCCESS; CloseRegionProcedure 3eae0a3c58bb2f53c1289450916a28ea, server=jenkins-hbase4.apache.org,35937,1676145619887 in 182 msec 2023-02-11 20:00:54,067 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=7c1f214213ec2e6ba08d051362b2ca4e, regionState=CLOSED 2023-02-11 20:00:54,067 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145654067"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145654067"}]},"ts":"1676145654067"} 2023-02-11 20:00:54,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:54,068 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=3518e490e7f4ccb7c60b36f15d4f9183, regionState=CLOSED 2023-02-11 20:00:54,068 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145654068"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145654068"}]},"ts":"1676145654068"} 2023-02-11 20:00:54,068 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, UNASSIGN in 215 msec 2023-02-11 20:00:54,069 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, UNASSIGN in 215 msec 2023-02-11 20:00:54,075 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=220 2023-02-11 20:00:54,075 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=220, state=SUCCESS; CloseRegionProcedure 38dcb256366159437c4db1abc8d1671c, server=jenkins-hbase4.apache.org,44175,1676145619924 in 200 msec 2023-02-11 20:00:54,085 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=217 2023-02-11 20:00:54,086 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=217, state=SUCCESS; CloseRegionProcedure 7c1f214213ec2e6ba08d051362b2ca4e, server=jenkins-hbase4.apache.org,35937,1676145619887 in 211 msec 2023-02-11 20:00:54,086 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, UNASSIGN in 224 msec 2023-02-11 20:00:54,087 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=218 2023-02-11 20:00:54,087 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=218, state=SUCCESS; CloseRegionProcedure 3518e490e7f4ccb7c60b36f15d4f9183, server=jenkins-hbase4.apache.org,44175,1676145619924 in 214 msec 2023-02-11 20:00:54,089 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, UNASSIGN in 234 msec 2023-02-11 20:00:54,092 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=218, resume processing ppid=215 2023-02-11 20:00:54,092 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, UNASSIGN in 236 msec 2023-02-11 20:00:54,093 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145654093"}]},"ts":"1676145654093"} 2023-02-11 20:00:54,095 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676145653041, state=DISABLED in hbase:meta 2023-02-11 20:00:54,097 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676145653041 to state=DISABLED 2023-02-11 20:00:54,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-11 20:00:54,100 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676145653041 in 256 msec 2023-02-11 20:00:54,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-11 20:00:54,600 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676145653041, procId: 215 completed 2023-02-11 20:00:54,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676145653041 table=testCorruptedSnapshot-1676145653041 type=FLUSH ttl=0 } 2023-02-11 20:00:54,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676145654600 (current time:1676145654600). 2023-02-11 20:00:54,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 20:00:54,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(139): Connect 0x23f6b24e to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:54,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6217679c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:54,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:54,610 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59500, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:54,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23f6b24e to 127.0.0.1:51732 2023-02-11 20:00:54,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:00:54,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 20:00:54,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676145653041 table=testCorruptedSnapshot-1676145653041 type=FLUSH ttl=0 } 2023-02-11 20:00:54,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676145653041, type=EXCLUSIVE 2023-02-11 20:00:54,613 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676145653041, type=EXCLUSIVE 2023-02-11 20:00:54,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676145653041 table=testCorruptedSnapshot-1676145653041 type=FLUSH ttl=0 } 2023-02-11 20:00:54,614 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676145653041, type=EXCLUSIVE; timeout=600000, timestamp=1676146254614 2023-02-11 20:00:54,614 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676145653041 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676145653041 2023-02-11 20:00:54,614 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676145653041 table=testCorruptedSnapshot-1676145653041 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 20:00:54,615 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 20:00:54,633 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676145653041 table=testCorruptedSnapshot-1676145653041 type=DISABLED ttl=0 } 2023-02-11 20:00:54,641 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:54,641 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:54,641 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:54,641 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:54,641 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:54,641 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:54,641 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:54,641 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:54,641 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:54,641 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:54,641 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:54,641 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:54,642 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:54,642 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:54,642 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:54,643 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:54,645 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:54,645 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:54,645 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:54,645 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:54,646 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 20:00:54,700 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676145653041 table=testCorruptedSnapshot-1676145653041 type=DISABLED ttl=0 } as finished. 2023-02-11 20:00:54,700 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676145653041 2023-02-11 20:00:54,701 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/emptySnaptb-1676145653041 2023-02-11 20:00:54,720 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/emptySnaptb-1676145653041 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/emptySnaptb-1676145653041 2023-02-11 20:00:54,735 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676145653041 of table testCorruptedSnapshot-1676145653041 completed 2023-02-11 20:00:54,735 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/emptySnaptb-1676145653041 2023-02-11 20:00:54,735 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/emptySnaptb-1676145653041 2023-02-11 20:00:54,735 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676145653041 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676145653041 at 1676145654614Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676145653041 table=testCorruptedSnapshot-1676145653041 type=DISABLED ttl=0 } at 1676145654633 (+19 ms)Consolidate snapshot: emptySnaptb-1676145653041 at 1676145654700 (+67 ms)Loading Region manifests for emptySnaptb-1676145653041 at 1676145654700Writing data manifest for emptySnaptb-1676145653041 at 1676145654706 (+6 ms)Verifying snapshot: emptySnaptb-1676145653041 at 1676145654715 (+9 ms)Snapshot emptySnaptb-1676145653041 of table testCorruptedSnapshot-1676145653041 completed at 1676145654735 (+20 ms) 2023-02-11 20:00:54,736 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676145653041, type=EXCLUSIVE 2023-02-11 20:00:54,737 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676145653041, type=EXCLUSIVE in 124 msec 2023-02-11 20:00:54,865 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 20:00:54,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676145653041 table=testCorruptedSnapshot-1676145653041 type=FLUSH ttl=0 } is done 2023-02-11 20:00:54,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676145653041 table=testCorruptedSnapshot-1676145653041 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 20:00:54,866 INFO [Listener at localhost/33691] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676145653041 2023-02-11 20:00:54,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676145653041 2023-02-11 20:00:54,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676145653041 2023-02-11 20:00:54,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-11 20:00:54,870 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676145653041 2023-02-11 20:00:54,870 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145654870"}]},"ts":"1676145654870"} 2023-02-11 20:00:54,871 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676145653041, state=ENABLING in hbase:meta 2023-02-11 20:00:54,874 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 20:00:54,874 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, ASSIGN}] 2023-02-11 20:00:54,874 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, ASSIGN 2023-02-11 20:00:54,875 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, ASSIGN 2023-02-11 20:00:54,875 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, ASSIGN 2023-02-11 20:00:54,875 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, ASSIGN 2023-02-11 20:00:54,875 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:54,875 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44175,1676145619924; forceNewPlan=false, retain=true 2023-02-11 20:00:54,875 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:54,875 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, ASSIGN 2023-02-11 20:00:54,875 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:54,875 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:54,875 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44175,1676145619924; forceNewPlan=false, retain=true 2023-02-11 20:00:54,875 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42011,1676145619841; forceNewPlan=false, retain=true 2023-02-11 20:00:54,876 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35937,1676145619887; forceNewPlan=false, retain=true 2023-02-11 20:00:54,876 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, ASSIGN 2023-02-11 20:00:54,876 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:54,876 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35937,1676145619887; forceNewPlan=false, retain=true 2023-02-11 20:00:54,876 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:54,876 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42011,1676145619841; forceNewPlan=false, retain=true 2023-02-11 20:00:55,025 INFO [jenkins-hbase4:39971] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 20:00:55,026 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=3518e490e7f4ccb7c60b36f15d4f9183, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:55,026 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=38dcb256366159437c4db1abc8d1671c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:55,026 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=5bc4c474972588bf6c06dc75b6e4aea6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:55,026 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145655026"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145655026"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145655026"}]},"ts":"1676145655026"} 2023-02-11 20:00:55,026 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=2f7f00f697d9b55aa03c57a3213d5a7c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:55,026 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=7c1f214213ec2e6ba08d051362b2ca4e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:55,026 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145655026"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145655026"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145655026"}]},"ts":"1676145655026"} 2023-02-11 20:00:55,026 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145655026"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145655026"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145655026"}]},"ts":"1676145655026"} 2023-02-11 20:00:55,026 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145655026"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145655026"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145655026"}]},"ts":"1676145655026"} 2023-02-11 20:00:55,026 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145655026"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145655026"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145655026"}]},"ts":"1676145655026"} 2023-02-11 20:00:55,028 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=234, state=RUNNABLE; OpenRegionProcedure 38dcb256366159437c4db1abc8d1671c, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:55,028 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=233, state=RUNNABLE; OpenRegionProcedure 2f7f00f697d9b55aa03c57a3213d5a7c, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:55,029 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=231, state=RUNNABLE; OpenRegionProcedure 7c1f214213ec2e6ba08d051362b2ca4e, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:55,029 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=230, state=RUNNABLE; OpenRegionProcedure 5bc4c474972588bf6c06dc75b6e4aea6, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:55,030 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=232, state=RUNNABLE; OpenRegionProcedure 3518e490e7f4ccb7c60b36f15d4f9183, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:55,030 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=3eae0a3c58bb2f53c1289450916a28ea, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:55,030 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145655030"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145655030"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145655030"}]},"ts":"1676145655030"} 2023-02-11 20:00:55,031 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=235, state=RUNNABLE; OpenRegionProcedure 3eae0a3c58bb2f53c1289450916a28ea, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:55,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-11 20:00:55,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:55,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 38dcb256366159437c4db1abc8d1671c, NAME => 'testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 20:00:55,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:55,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:55,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:55,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:55,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5bc4c474972588bf6c06dc75b6e4aea6, NAME => 'testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 20:00:55,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7c1f214213ec2e6ba08d051362b2ca4e, NAME => 'testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 20:00:55,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:55,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:55,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:55,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:55,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:55,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:55,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:55,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:55,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:55,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:55,184 INFO [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:55,188 INFO [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:55,188 INFO [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:55,188 DEBUG [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf 2023-02-11 20:00:55,188 DEBUG [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf 2023-02-11 20:00:55,188 DEBUG [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf 2023-02-11 20:00:55,188 DEBUG [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf 2023-02-11 20:00:55,188 INFO [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 38dcb256366159437c4db1abc8d1671c columnFamilyName cf 2023-02-11 20:00:55,188 DEBUG [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf 2023-02-11 20:00:55,189 DEBUG [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf 2023-02-11 20:00:55,189 INFO [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7c1f214213ec2e6ba08d051362b2ca4e columnFamilyName cf 2023-02-11 20:00:55,189 INFO [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5bc4c474972588bf6c06dc75b6e4aea6 columnFamilyName cf 2023-02-11 20:00:55,189 INFO [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] regionserver.HStore(310): Store=38dcb256366159437c4db1abc8d1671c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:55,189 INFO [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] regionserver.HStore(310): Store=7c1f214213ec2e6ba08d051362b2ca4e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:55,190 INFO [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] regionserver.HStore(310): Store=5bc4c474972588bf6c06dc75b6e4aea6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:55,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:55,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:55,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:55,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:55,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:55,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:55,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:55,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:55,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:55,194 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 38dcb256366159437c4db1abc8d1671c; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74621733, jitterRate=0.11195047199726105}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:55,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 38dcb256366159437c4db1abc8d1671c: Running coprocessor pre-open hook at 1676145655183Writing region info on filesystem at 1676145655183Initializing all the Stores at 1676145655183Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145655183Cleaning up temporary data from old regions at 1676145655191 (+8 ms)Cleaning up detritus from prior splits at 1676145655192 (+1 ms)Running coprocessor post-open hooks at 1676145655194 (+2 ms)Region opened successfully at 1676145655194 2023-02-11 20:00:55,194 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7c1f214213ec2e6ba08d051362b2ca4e; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74702584, jitterRate=0.11315524578094482}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:55,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7c1f214213ec2e6ba08d051362b2ca4e: Running coprocessor pre-open hook at 1676145655183Writing region info on filesystem at 1676145655183Initializing all the Stores at 1676145655184 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145655184Cleaning up temporary data from old regions at 1676145655191 (+7 ms)Cleaning up detritus from prior splits at 1676145655192 (+1 ms)Running coprocessor post-open hooks at 1676145655194 (+2 ms)Region opened successfully at 1676145655194 2023-02-11 20:00:55,194 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5bc4c474972588bf6c06dc75b6e4aea6; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74581823, jitterRate=0.11135576665401459}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:55,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5bc4c474972588bf6c06dc75b6e4aea6: Running coprocessor pre-open hook at 1676145655183Writing region info on filesystem at 1676145655183Initializing all the Stores at 1676145655184 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145655184Cleaning up temporary data from old regions at 1676145655191 (+7 ms)Cleaning up detritus from prior splits at 1676145655192 (+1 ms)Running coprocessor post-open hooks at 1676145655194 (+2 ms)Region opened successfully at 1676145655194 2023-02-11 20:00:55,195 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e., pid=238, masterSystemTime=1676145655180 2023-02-11 20:00:55,195 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c., pid=236, masterSystemTime=1676145655179 2023-02-11 20:00:55,195 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6., pid=239, masterSystemTime=1676145655180 2023-02-11 20:00:55,196 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:55,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:55,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:55,196 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3eae0a3c58bb2f53c1289450916a28ea, NAME => 'testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 20:00:55,196 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:55,197 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:55,197 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:55,197 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:55,197 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=7c1f214213ec2e6ba08d051362b2ca4e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:55,197 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145655197"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145655197"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145655197"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145655197"}]},"ts":"1676145655197"} 2023-02-11 20:00:55,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:55,198 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:55,198 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:55,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3518e490e7f4ccb7c60b36f15d4f9183, NAME => 'testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 20:00:55,199 INFO [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:55,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:55,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:55,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:55,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:55,200 DEBUG [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf 2023-02-11 20:00:55,200 DEBUG [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf 2023-02-11 20:00:55,200 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=38dcb256366159437c4db1abc8d1671c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:55,200 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145655200"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145655200"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145655200"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145655200"}]},"ts":"1676145655200"} 2023-02-11 20:00:55,200 INFO [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3eae0a3c58bb2f53c1289450916a28ea columnFamilyName cf 2023-02-11 20:00:55,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:55,201 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:55,201 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:55,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2f7f00f697d9b55aa03c57a3213d5a7c, NAME => 'testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 20:00:55,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:55,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:55,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:55,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:55,201 INFO [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:55,202 INFO [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] regionserver.HStore(310): Store=3eae0a3c58bb2f53c1289450916a28ea/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:55,202 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=5bc4c474972588bf6c06dc75b6e4aea6, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:55,202 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145655202"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145655202"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145655202"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145655202"}]},"ts":"1676145655202"} 2023-02-11 20:00:55,202 INFO [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:55,203 DEBUG [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf 2023-02-11 20:00:55,203 DEBUG [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf 2023-02-11 20:00:55,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:55,203 DEBUG [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf 2023-02-11 20:00:55,203 DEBUG [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf 2023-02-11 20:00:55,203 INFO [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3518e490e7f4ccb7c60b36f15d4f9183 columnFamilyName cf 2023-02-11 20:00:55,203 INFO [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2f7f00f697d9b55aa03c57a3213d5a7c columnFamilyName cf 2023-02-11 20:00:55,204 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=231 2023-02-11 20:00:55,204 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=231, state=SUCCESS; OpenRegionProcedure 7c1f214213ec2e6ba08d051362b2ca4e, server=jenkins-hbase4.apache.org,35937,1676145619887 in 171 msec 2023-02-11 20:00:55,204 INFO [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] regionserver.HStore(310): Store=3518e490e7f4ccb7c60b36f15d4f9183/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:55,204 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:55,205 INFO [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] regionserver.HStore(310): Store=2f7f00f697d9b55aa03c57a3213d5a7c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:55,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:55,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:55,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:55,207 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, ASSIGN in 330 msec 2023-02-11 20:00:55,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:55,207 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=234 2023-02-11 20:00:55,207 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=234, state=SUCCESS; OpenRegionProcedure 38dcb256366159437c4db1abc8d1671c, server=jenkins-hbase4.apache.org,44175,1676145619924 in 176 msec 2023-02-11 20:00:55,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:55,209 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=230 2023-02-11 20:00:55,209 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, ASSIGN in 333 msec 2023-02-11 20:00:55,209 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=230, state=SUCCESS; OpenRegionProcedure 5bc4c474972588bf6c06dc75b6e4aea6, server=jenkins-hbase4.apache.org,42011,1676145619841 in 176 msec 2023-02-11 20:00:55,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:55,210 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3518e490e7f4ccb7c60b36f15d4f9183; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59852871, jitterRate=-0.10812272131443024}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:55,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3518e490e7f4ccb7c60b36f15d4f9183: Running coprocessor pre-open hook at 1676145655199Writing region info on filesystem at 1676145655199Initializing all the Stores at 1676145655201 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145655201Cleaning up temporary data from old regions at 1676145655206 (+5 ms)Cleaning up detritus from prior splits at 1676145655207 (+1 ms)Running coprocessor post-open hooks at 1676145655210 (+3 ms)Region opened successfully at 1676145655210 2023-02-11 20:00:55,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:55,210 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3eae0a3c58bb2f53c1289450916a28ea; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71215944, jitterRate=0.06120026111602783}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:55,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3eae0a3c58bb2f53c1289450916a28ea: Running coprocessor pre-open hook at 1676145655197Writing region info on filesystem at 1676145655197Initializing all the Stores at 1676145655198 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145655198Cleaning up temporary data from old regions at 1676145655205 (+7 ms)Cleaning up detritus from prior splits at 1676145655206 (+1 ms)Running coprocessor post-open hooks at 1676145655210 (+4 ms)Region opened successfully at 1676145655210 2023-02-11 20:00:55,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2f7f00f697d9b55aa03c57a3213d5a7c; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70381239, jitterRate=0.048762187361717224}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:55,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea., pid=241, masterSystemTime=1676145655180 2023-02-11 20:00:55,211 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, ASSIGN in 335 msec 2023-02-11 20:00:55,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183., pid=240, masterSystemTime=1676145655179 2023-02-11 20:00:55,211 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2f7f00f697d9b55aa03c57a3213d5a7c: Running coprocessor pre-open hook at 1676145655201Writing region info on filesystem at 1676145655201Initializing all the Stores at 1676145655202 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145655202Cleaning up temporary data from old regions at 1676145655208 (+6 ms)Cleaning up detritus from prior splits at 1676145655208Running coprocessor post-open hooks at 1676145655211 (+3 ms)Region opened successfully at 1676145655211 2023-02-11 20:00:55,212 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c., pid=237, masterSystemTime=1676145655180 2023-02-11 20:00:55,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:55,212 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:55,213 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=3eae0a3c58bb2f53c1289450916a28ea, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:55,213 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145655213"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145655213"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145655213"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145655213"}]},"ts":"1676145655213"} 2023-02-11 20:00:55,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:55,213 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:55,214 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=3518e490e7f4ccb7c60b36f15d4f9183, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:55,214 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145655214"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145655214"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145655214"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145655214"}]},"ts":"1676145655214"} 2023-02-11 20:00:55,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:55,215 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:55,215 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=2f7f00f697d9b55aa03c57a3213d5a7c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:55,215 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145655215"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145655215"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145655215"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145655215"}]},"ts":"1676145655215"} 2023-02-11 20:00:55,219 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=235 2023-02-11 20:00:55,219 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=235, state=SUCCESS; OpenRegionProcedure 3eae0a3c58bb2f53c1289450916a28ea, server=jenkins-hbase4.apache.org,35937,1676145619887 in 184 msec 2023-02-11 20:00:55,220 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=232 2023-02-11 20:00:55,220 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=232, state=SUCCESS; OpenRegionProcedure 3518e490e7f4ccb7c60b36f15d4f9183, server=jenkins-hbase4.apache.org,44175,1676145619924 in 186 msec 2023-02-11 20:00:55,221 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, ASSIGN in 345 msec 2023-02-11 20:00:55,221 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=233 2023-02-11 20:00:55,221 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=233, state=SUCCESS; OpenRegionProcedure 2f7f00f697d9b55aa03c57a3213d5a7c, server=jenkins-hbase4.apache.org,42011,1676145619841 in 191 msec 2023-02-11 20:00:55,222 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, ASSIGN in 346 msec 2023-02-11 20:00:55,223 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=233, resume processing ppid=229 2023-02-11 20:00:55,223 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, ASSIGN in 347 msec 2023-02-11 20:00:55,224 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145655223"}]},"ts":"1676145655223"} 2023-02-11 20:00:55,225 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676145653041, state=ENABLED in hbase:meta 2023-02-11 20:00:55,226 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676145653041' was successfully enabled. 2023-02-11 20:00:55,228 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676145653041 in 360 msec 2023-02-11 20:00:55,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-11 20:00:55,622 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676145653041, procId: 229 completed 2023-02-11 20:00:55,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:55,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:55,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:55,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:55,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:55,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:55,650 DEBUG [Listener at localhost/33691] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676145653041 2023-02-11 20:00:55,650 DEBUG [Listener at localhost/33691] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:55,650 INFO [Listener at localhost/33691] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 20:00:55,665 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 20:00:55,666 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 20:00:55,667 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 20:00:55,668 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 20:00:55,670 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 20:00:55,673 INFO [Listener at localhost/33691] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676145653041 2023-02-11 20:00:55,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676145653041 2023-02-11 20:00:55,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676145653041 2023-02-11 20:00:55,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-11 20:00:55,678 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145655678"}]},"ts":"1676145655678"} 2023-02-11 20:00:55,680 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676145653041, state=DISABLING in hbase:meta 2023-02-11 20:00:55,683 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676145653041 to state=DISABLING 2023-02-11 20:00:55,683 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, UNASSIGN}] 2023-02-11 20:00:55,684 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, UNASSIGN 2023-02-11 20:00:55,685 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, UNASSIGN 2023-02-11 20:00:55,685 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, UNASSIGN 2023-02-11 20:00:55,685 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, UNASSIGN 2023-02-11 20:00:55,685 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, UNASSIGN 2023-02-11 20:00:55,686 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=2f7f00f697d9b55aa03c57a3213d5a7c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:55,686 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145655685"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145655685"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145655685"}]},"ts":"1676145655685"} 2023-02-11 20:00:55,686 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=3518e490e7f4ccb7c60b36f15d4f9183, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:55,686 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=3eae0a3c58bb2f53c1289450916a28ea, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:55,686 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=7c1f214213ec2e6ba08d051362b2ca4e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:55,686 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145655686"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145655686"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145655686"}]},"ts":"1676145655686"} 2023-02-11 20:00:55,686 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145655686"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145655686"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145655686"}]},"ts":"1676145655686"} 2023-02-11 20:00:55,686 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145655686"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145655686"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145655686"}]},"ts":"1676145655686"} 2023-02-11 20:00:55,686 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=5bc4c474972588bf6c06dc75b6e4aea6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:55,687 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145655686"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145655686"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145655686"}]},"ts":"1676145655686"} 2023-02-11 20:00:55,688 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=246, state=RUNNABLE; CloseRegionProcedure 2f7f00f697d9b55aa03c57a3213d5a7c, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:55,688 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=248, state=RUNNABLE; CloseRegionProcedure 3eae0a3c58bb2f53c1289450916a28ea, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:55,689 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=244, state=RUNNABLE; CloseRegionProcedure 7c1f214213ec2e6ba08d051362b2ca4e, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:55,690 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=245, state=RUNNABLE; CloseRegionProcedure 3518e490e7f4ccb7c60b36f15d4f9183, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:55,693 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=243, state=RUNNABLE; CloseRegionProcedure 5bc4c474972588bf6c06dc75b6e4aea6, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:55,694 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, UNASSIGN 2023-02-11 20:00:55,696 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=38dcb256366159437c4db1abc8d1671c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:55,696 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145655696"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145655696"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145655696"}]},"ts":"1676145655696"} 2023-02-11 20:00:55,704 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=247, state=RUNNABLE; CloseRegionProcedure 38dcb256366159437c4db1abc8d1671c, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:55,842 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:55,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2f7f00f697d9b55aa03c57a3213d5a7c, disabling compactions & flushes 2023-02-11 20:00:55,842 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:55,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:55,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. after waiting 0 ms 2023-02-11 20:00:55,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:55,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 2f7f00f697d9b55aa03c57a3213d5a7c 1/1 column families, dataSize=2.29 KB heapSize=5.17 KB 2023-02-11 20:00:55,842 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:55,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3eae0a3c58bb2f53c1289450916a28ea, disabling compactions & flushes 2023-02-11 20:00:55,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:55,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:55,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. after waiting 0 ms 2023-02-11 20:00:55,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:55,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 3eae0a3c58bb2f53c1289450916a28ea 1/1 column families, dataSize=22.57 KB heapSize=48.77 KB 2023-02-11 20:00:55,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:55,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 38dcb256366159437c4db1abc8d1671c, disabling compactions & flushes 2023-02-11 20:00:55,846 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:55,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:55,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. after waiting 0 ms 2023-02-11 20:00:55,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:55,846 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 38dcb256366159437c4db1abc8d1671c 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-11 20:00:55,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-11 20:00:55,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302117a42971387dc4e4abe5be69050d0dbca to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/e4da3b7fbbce2345d7772b0674a318d5202302117a42971387dc4e4abe5be69050d0dbca 2023-02-11 20:00:55,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230211dcc9bcc5d07b49e88321a7e2f185f411 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/a87ff679a2f3e71d9181a67b7542122c20230211dcc9bcc5d07b49e88321a7e2f185f411 2023-02-11 20:00:55,947 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230211f8be42215c3f4b14923eab8a227b17a6 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211f8be42215c3f4b14923eab8a227b17a6 2023-02-11 20:00:55,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/.tmp/cf/c59ef0e4ab8e4494b8187a90d196014d 2023-02-11 20:00:55,953 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/.tmp/cf/f549005d553c4c269a0d1574c1fcb42b 2023-02-11 20:00:55,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/.tmp/cf/f549005d553c4c269a0d1574c1fcb42b as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf/f549005d553c4c269a0d1574c1fcb42b 2023-02-11 20:00:55,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/.tmp/cf/c59ef0e4ab8e4494b8187a90d196014d as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf/c59ef0e4ab8e4494b8187a90d196014d 2023-02-11 20:00:55,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf/c59ef0e4ab8e4494b8187a90d196014d, entries=345, sequenceid=8, filesize=64.4 K 2023-02-11 20:00:55,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf/f549005d553c4c269a0d1574c1fcb42b, entries=36, sequenceid=8, filesize=11.1 K 2023-02-11 20:00:55,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.57 KB/23111, heapSize ~48.75 KB/49920, currentSize=0 B/0 for 3eae0a3c58bb2f53c1289450916a28ea in 119ms, sequenceid=8, compaction requested=false 2023-02-11 20:00:55,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 38dcb256366159437c4db1abc8d1671c in 116ms, sequenceid=8, compaction requested=false 2023-02-11 20:00:55,962 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676145653041' 2023-02-11 20:00:55,962 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676145653041' 2023-02-11 20:00:56,000 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/.tmp/cf/726be4c9648c48bf9dccefde2083eb5e 2023-02-11 20:00:56,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/.tmp/cf/726be4c9648c48bf9dccefde2083eb5e as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf/726be4c9648c48bf9dccefde2083eb5e 2023-02-11 20:00:56,008 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 20:00:56,009 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:56,009 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf/726be4c9648c48bf9dccefde2083eb5e, entries=35, sequenceid=8, filesize=10.9 K 2023-02-11 20:00:56,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 38dcb256366159437c4db1abc8d1671c: Waiting for close lock at 1676145655845Running coprocessor pre-close hooks at 1676145655845Disabling compacts and flushes for region at 1676145655845Disabling writes for close at 1676145655846 (+1 ms)Obtaining lock to block concurrent updates at 1676145655846Preparing flush snapshotting stores in 38dcb256366159437c4db1abc8d1671c at 1676145655846Finished memstore snapshotting testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676145655846Flushing stores of testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. at 1676145655848 (+2 ms)Flushing 38dcb256366159437c4db1abc8d1671c/cf: creating writer at 1676145655849 (+1 ms)Flushing mob file 38dcb256366159437c4db1abc8d1671c/cf: appending metadata at 1676145655873 (+24 ms)Flushing mob file 38dcb256366159437c4db1abc8d1671c/cf: closing flushed file at 1676145655873Flushing 38dcb256366159437c4db1abc8d1671c/cf: appending metadata at 1676145655942 (+69 ms)Flushing 38dcb256366159437c4db1abc8d1671c/cf: closing flushed file at 1676145655942Flushing 38dcb256366159437c4db1abc8d1671c/cf: reopening flushed file at 1676145655958 (+16 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 38dcb256366159437c4db1abc8d1671c in 116ms, sequenceid=8, compaction requested=false at 1676145655962 (+4 ms)Writing region close event to WAL at 1676145655993 (+31 ms)Running coprocessor post-close hooks at 1676145656009 (+16 ms)Closed at 1676145656009 2023-02-11 20:00:56,011 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for 2f7f00f697d9b55aa03c57a3213d5a7c in 168ms, sequenceid=8, compaction requested=false 2023-02-11 20:00:56,012 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:56,012 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:56,012 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3518e490e7f4ccb7c60b36f15d4f9183, disabling compactions & flushes 2023-02-11 20:00:56,012 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:56,012 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=38dcb256366159437c4db1abc8d1671c, regionState=CLOSED 2023-02-11 20:00:56,012 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:56,013 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. after waiting 0 ms 2023-02-11 20:00:56,013 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:56,013 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145656012"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145656012"}]},"ts":"1676145656012"} 2023-02-11 20:00:56,013 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 3518e490e7f4ccb7c60b36f15d4f9183 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-11 20:00:56,014 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 20:00:56,014 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:56,014 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3eae0a3c58bb2f53c1289450916a28ea: Waiting for close lock at 1676145655843Running coprocessor pre-close hooks at 1676145655843Disabling compacts and flushes for region at 1676145655843Disabling writes for close at 1676145655843Obtaining lock to block concurrent updates at 1676145655843Preparing flush snapshotting stores in 3eae0a3c58bb2f53c1289450916a28ea at 1676145655843Finished memstore snapshotting testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea., syncing WAL and waiting on mvcc, flushsize=dataSize=23111, getHeapSize=49920, getOffHeapSize=0, getCellsCount=345 at 1676145655844 (+1 ms)Flushing stores of testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. at 1676145655846 (+2 ms)Flushing 3eae0a3c58bb2f53c1289450916a28ea/cf: creating writer at 1676145655847 (+1 ms)Flushing mob file 3eae0a3c58bb2f53c1289450916a28ea/cf: appending metadata at 1676145655875 (+28 ms)Flushing mob file 3eae0a3c58bb2f53c1289450916a28ea/cf: closing flushed file at 1676145655875Flushing 3eae0a3c58bb2f53c1289450916a28ea/cf: appending metadata at 1676145655942 (+67 ms)Flushing 3eae0a3c58bb2f53c1289450916a28ea/cf: closing flushed file at 1676145655942Flushing 3eae0a3c58bb2f53c1289450916a28ea/cf: reopening flushed file at 1676145655958 (+16 ms)Finished flush of dataSize ~22.57 KB/23111, heapSize ~48.75 KB/49920, currentSize=0 B/0 for 3eae0a3c58bb2f53c1289450916a28ea in 119ms, sequenceid=8, compaction requested=false at 1676145655962 (+4 ms)Writing region close event to WAL at 1676145655993 (+31 ms)Running coprocessor post-close hooks at 1676145656014 (+21 ms)Closed at 1676145656014 2023-02-11 20:00:56,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=247 2023-02-11 20:00:56,016 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=3eae0a3c58bb2f53c1289450916a28ea, regionState=CLOSED 2023-02-11 20:00:56,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:56,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:56,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7c1f214213ec2e6ba08d051362b2ca4e, disabling compactions & flushes 2023-02-11 20:00:56,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:56,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:56,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. after waiting 0 ms 2023-02-11 20:00:56,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:56,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7c1f214213ec2e6ba08d051362b2ca4e 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-11 20:00:56,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=247, state=SUCCESS; CloseRegionProcedure 38dcb256366159437c4db1abc8d1671c, server=jenkins-hbase4.apache.org,44175,1676145619924 in 310 msec 2023-02-11 20:00:56,016 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145656016"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145656016"}]},"ts":"1676145656016"} 2023-02-11 20:00:56,018 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, UNASSIGN in 333 msec 2023-02-11 20:00:56,030 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=248 2023-02-11 20:00:56,030 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=248, state=SUCCESS; CloseRegionProcedure 3eae0a3c58bb2f53c1289450916a28ea, server=jenkins-hbase4.apache.org,35937,1676145619887 in 331 msec 2023-02-11 20:00:56,031 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 20:00:56,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:56,033 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, UNASSIGN in 347 msec 2023-02-11 20:00:56,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2f7f00f697d9b55aa03c57a3213d5a7c: Waiting for close lock at 1676145655842Running coprocessor pre-close hooks at 1676145655842Disabling compacts and flushes for region at 1676145655842Disabling writes for close at 1676145655843 (+1 ms)Obtaining lock to block concurrent updates at 1676145655843Preparing flush snapshotting stores in 2f7f00f697d9b55aa03c57a3213d5a7c at 1676145655843Finished memstore snapshotting testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c., syncing WAL and waiting on mvcc, flushsize=dataSize=2343, getHeapSize=5280, getOffHeapSize=0, getCellsCount=35 at 1676145655843Flushing stores of testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. at 1676145655844 (+1 ms)Flushing 2f7f00f697d9b55aa03c57a3213d5a7c/cf: creating writer at 1676145655845 (+1 ms)Flushing mob file 2f7f00f697d9b55aa03c57a3213d5a7c/cf: appending metadata at 1676145655872 (+27 ms)Flushing mob file 2f7f00f697d9b55aa03c57a3213d5a7c/cf: closing flushed file at 1676145655872Flushing 2f7f00f697d9b55aa03c57a3213d5a7c/cf: appending metadata at 1676145655948 (+76 ms)Flushing 2f7f00f697d9b55aa03c57a3213d5a7c/cf: closing flushed file at 1676145655948Flushing 2f7f00f697d9b55aa03c57a3213d5a7c/cf: reopening flushed file at 1676145656005 (+57 ms)Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for 2f7f00f697d9b55aa03c57a3213d5a7c in 168ms, sequenceid=8, compaction requested=false at 1676145656011 (+6 ms)Writing region close event to WAL at 1676145656027 (+16 ms)Running coprocessor post-close hooks at 1676145656032 (+5 ms)Closed at 1676145656032 2023-02-11 20:00:56,035 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:56,036 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:56,036 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5bc4c474972588bf6c06dc75b6e4aea6, disabling compactions & flushes 2023-02-11 20:00:56,036 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:56,036 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:56,036 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. after waiting 0 ms 2023-02-11 20:00:56,036 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:56,036 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=2f7f00f697d9b55aa03c57a3213d5a7c, regionState=CLOSED 2023-02-11 20:00:56,036 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145656036"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145656036"}]},"ts":"1676145656036"} 2023-02-11 20:00:56,036 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 5bc4c474972588bf6c06dc75b6e4aea6 1/1 column families, dataSize=1.63 KB heapSize=3.77 KB 2023-02-11 20:00:56,040 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=246 2023-02-11 20:00:56,040 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=246, state=SUCCESS; CloseRegionProcedure 2f7f00f697d9b55aa03c57a3213d5a7c, server=jenkins-hbase4.apache.org,42011,1676145619841 in 350 msec 2023-02-11 20:00:56,041 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, UNASSIGN in 357 msec 2023-02-11 20:00:56,097 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230211bee74755269547a78094d6514813ce6e to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c4ca4238a0b923820dcc509a6f75849b20230211bee74755269547a78094d6514813ce6e 2023-02-11 20:00:56,097 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230211847278d687ef4e8094a419ca4751f0b4 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c81e728d9d4c2f636f067f89cc14862c20230211847278d687ef4e8094a419ca4751f0b4 2023-02-11 20:00:56,132 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230211a615ab38ba894a7bb51b2d27eb43d491 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/d41d8cd98f00b204e9800998ecf8427e20230211a615ab38ba894a7bb51b2d27eb43d491 2023-02-11 20:00:56,168 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/.tmp/cf/8f8d90d1d9fe4323a8458e43b284b07f 2023-02-11 20:00:56,174 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/.tmp/cf/8f8d90d1d9fe4323a8458e43b284b07f as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf/8f8d90d1d9fe4323a8458e43b284b07f 2023-02-11 20:00:56,175 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/.tmp/cf/e0b7ecd250ec4bd99377381ecb2a6f2a 2023-02-11 20:00:56,178 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/.tmp/cf/583edb103c644b3182f51b355c9fe11d 2023-02-11 20:00:56,180 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/.tmp/cf/e0b7ecd250ec4bd99377381ecb2a6f2a as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf/e0b7ecd250ec4bd99377381ecb2a6f2a 2023-02-11 20:00:56,181 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf/8f8d90d1d9fe4323a8458e43b284b07f, entries=30, sequenceid=8, filesize=10.1 K 2023-02-11 20:00:56,183 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 3518e490e7f4ccb7c60b36f15d4f9183 in 170ms, sequenceid=8, compaction requested=false 2023-02-11 20:00:56,183 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/.tmp/cf/583edb103c644b3182f51b355c9fe11d as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf/583edb103c644b3182f51b355c9fe11d 2023-02-11 20:00:56,187 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf/e0b7ecd250ec4bd99377381ecb2a6f2a, entries=29, sequenceid=8, filesize=9.9 K 2023-02-11 20:00:56,188 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 7c1f214213ec2e6ba08d051362b2ca4e in 171ms, sequenceid=8, compaction requested=false 2023-02-11 20:00:56,191 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf/583edb103c644b3182f51b355c9fe11d, entries=25, sequenceid=8, filesize=9.2 K 2023-02-11 20:00:56,191 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.63 KB/1673, heapSize ~3.75 KB/3840, currentSize=0 B/0 for 5bc4c474972588bf6c06dc75b6e4aea6 in 155ms, sequenceid=8, compaction requested=false 2023-02-11 20:00:56,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 20:00:56,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:56,223 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7c1f214213ec2e6ba08d051362b2ca4e: Waiting for close lock at 1676145656017Running coprocessor pre-close hooks at 1676145656017Disabling compacts and flushes for region at 1676145656017Disabling writes for close at 1676145656017Obtaining lock to block concurrent updates at 1676145656017Preparing flush snapshotting stores in 7c1f214213ec2e6ba08d051362b2ca4e at 1676145656017Finished memstore snapshotting testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e., syncing WAL and waiting on mvcc, flushsize=dataSize=1941, getHeapSize=4416, getOffHeapSize=0, getCellsCount=29 at 1676145656017Flushing stores of testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. at 1676145656018 (+1 ms)Flushing 7c1f214213ec2e6ba08d051362b2ca4e/cf: creating writer at 1676145656027 (+9 ms)Flushing mob file 7c1f214213ec2e6ba08d051362b2ca4e/cf: appending metadata at 1676145656042 (+15 ms)Flushing mob file 7c1f214213ec2e6ba08d051362b2ca4e/cf: closing flushed file at 1676145656042Flushing 7c1f214213ec2e6ba08d051362b2ca4e/cf: appending metadata at 1676145656099 (+57 ms)Flushing 7c1f214213ec2e6ba08d051362b2ca4e/cf: closing flushed file at 1676145656099Flushing 7c1f214213ec2e6ba08d051362b2ca4e/cf: reopening flushed file at 1676145656181 (+82 ms)Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 7c1f214213ec2e6ba08d051362b2ca4e in 171ms, sequenceid=8, compaction requested=false at 1676145656188 (+7 ms)Writing region close event to WAL at 1676145656218 (+30 ms)Running coprocessor post-close hooks at 1676145656223 (+5 ms)Closed at 1676145656223 2023-02-11 20:00:56,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 20:00:56,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:56,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5bc4c474972588bf6c06dc75b6e4aea6: Waiting for close lock at 1676145656036Running coprocessor pre-close hooks at 1676145656036Disabling compacts and flushes for region at 1676145656036Disabling writes for close at 1676145656036Obtaining lock to block concurrent updates at 1676145656036Preparing flush snapshotting stores in 5bc4c474972588bf6c06dc75b6e4aea6 at 1676145656036Finished memstore snapshotting testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6., syncing WAL and waiting on mvcc, flushsize=dataSize=1673, getHeapSize=3840, getOffHeapSize=0, getCellsCount=25 at 1676145656037 (+1 ms)Flushing stores of testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. at 1676145656037Flushing 5bc4c474972588bf6c06dc75b6e4aea6/cf: creating writer at 1676145656037Flushing mob file 5bc4c474972588bf6c06dc75b6e4aea6/cf: appending metadata at 1676145656044 (+7 ms)Flushing mob file 5bc4c474972588bf6c06dc75b6e4aea6/cf: closing flushed file at 1676145656044Flushing 5bc4c474972588bf6c06dc75b6e4aea6/cf: appending metadata at 1676145656133 (+89 ms)Flushing 5bc4c474972588bf6c06dc75b6e4aea6/cf: closing flushed file at 1676145656133Flushing 5bc4c474972588bf6c06dc75b6e4aea6/cf: reopening flushed file at 1676145656184 (+51 ms)Finished flush of dataSize ~1.63 KB/1673, heapSize ~3.75 KB/3840, currentSize=0 B/0 for 5bc4c474972588bf6c06dc75b6e4aea6 in 155ms, sequenceid=8, compaction requested=false at 1676145656191 (+7 ms)Writing region close event to WAL at 1676145656218 (+27 ms)Running coprocessor post-close hooks at 1676145656224 (+6 ms)Closed at 1676145656224 2023-02-11 20:00:56,225 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 20:00:56,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:56,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:56,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3518e490e7f4ccb7c60b36f15d4f9183: Waiting for close lock at 1676145656012Running coprocessor pre-close hooks at 1676145656012Disabling compacts and flushes for region at 1676145656012Disabling writes for close at 1676145656013 (+1 ms)Obtaining lock to block concurrent updates at 1676145656013Preparing flush snapshotting stores in 3518e490e7f4ccb7c60b36f15d4f9183 at 1676145656013Finished memstore snapshotting testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676145656013Flushing stores of testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. at 1676145656014 (+1 ms)Flushing 3518e490e7f4ccb7c60b36f15d4f9183/cf: creating writer at 1676145656027 (+13 ms)Flushing mob file 3518e490e7f4ccb7c60b36f15d4f9183/cf: appending metadata at 1676145656031 (+4 ms)Flushing mob file 3518e490e7f4ccb7c60b36f15d4f9183/cf: closing flushed file at 1676145656031Flushing 3518e490e7f4ccb7c60b36f15d4f9183/cf: appending metadata at 1676145656099 (+68 ms)Flushing 3518e490e7f4ccb7c60b36f15d4f9183/cf: closing flushed file at 1676145656099Flushing 3518e490e7f4ccb7c60b36f15d4f9183/cf: reopening flushed file at 1676145656176 (+77 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 3518e490e7f4ccb7c60b36f15d4f9183 in 170ms, sequenceid=8, compaction requested=false at 1676145656183 (+7 ms)Writing region close event to WAL at 1676145656218 (+35 ms)Running coprocessor post-close hooks at 1676145656226 (+8 ms)Closed at 1676145656226 2023-02-11 20:00:56,226 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=7c1f214213ec2e6ba08d051362b2ca4e, regionState=CLOSED 2023-02-11 20:00:56,227 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145656226"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145656226"}]},"ts":"1676145656226"} 2023-02-11 20:00:56,227 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:56,227 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=5bc4c474972588bf6c06dc75b6e4aea6, regionState=CLOSED 2023-02-11 20:00:56,228 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145656227"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145656227"}]},"ts":"1676145656227"} 2023-02-11 20:00:56,228 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:56,228 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=3518e490e7f4ccb7c60b36f15d4f9183, regionState=CLOSED 2023-02-11 20:00:56,228 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145656228"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145656228"}]},"ts":"1676145656228"} 2023-02-11 20:00:56,233 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=244 2023-02-11 20:00:56,233 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=244, state=SUCCESS; CloseRegionProcedure 7c1f214213ec2e6ba08d051362b2ca4e, server=jenkins-hbase4.apache.org,35937,1676145619887 in 540 msec 2023-02-11 20:00:56,238 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=243 2023-02-11 20:00:56,238 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, UNASSIGN in 550 msec 2023-02-11 20:00:56,238 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=245 2023-02-11 20:00:56,238 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=243, state=SUCCESS; CloseRegionProcedure 5bc4c474972588bf6c06dc75b6e4aea6, server=jenkins-hbase4.apache.org,42011,1676145619841 in 537 msec 2023-02-11 20:00:56,238 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=245, state=SUCCESS; CloseRegionProcedure 3518e490e7f4ccb7c60b36f15d4f9183, server=jenkins-hbase4.apache.org,44175,1676145619924 in 542 msec 2023-02-11 20:00:56,240 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, UNASSIGN in 555 msec 2023-02-11 20:00:56,242 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=245, resume processing ppid=242 2023-02-11 20:00:56,242 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, UNASSIGN in 555 msec 2023-02-11 20:00:56,242 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145656242"}]},"ts":"1676145656242"} 2023-02-11 20:00:56,245 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676145653041, state=DISABLED in hbase:meta 2023-02-11 20:00:56,247 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676145653041 to state=DISABLED 2023-02-11 20:00:56,250 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676145653041 in 575 msec 2023-02-11 20:00:56,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-11 20:00:56,434 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676145653041, procId: 242 completed 2023-02-11 20:00:56,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676145653041 table=testCorruptedSnapshot-1676145653041 type=FLUSH ttl=0 } 2023-02-11 20:00:56,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676145656436 (current time:1676145656436). 2023-02-11 20:00:56,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 20:00:56,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(139): Connect 0x58de38ea to 127.0.0.1:51732 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 20:00:56,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4eac957d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 20:00:56,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 20:00:56,450 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59504, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 20:00:56,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58de38ea to 127.0.0.1:51732 2023-02-11 20:00:56,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:00:56,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 20:00:56,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676145653041 table=testCorruptedSnapshot-1676145653041 type=FLUSH ttl=0 } 2023-02-11 20:00:56,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676145653041, type=EXCLUSIVE 2023-02-11 20:00:56,456 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676145653041, type=EXCLUSIVE 2023-02-11 20:00:56,464 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676145653041, type=EXCLUSIVE; timeout=600000, timestamp=1676146256464 2023-02-11 20:00:56,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676145653041 table=testCorruptedSnapshot-1676145653041 type=FLUSH ttl=0 } 2023-02-11 20:00:56,464 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676145653041 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676145653041 2023-02-11 20:00:56,465 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676145653041 table=testCorruptedSnapshot-1676145653041 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 20:00:56,465 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 20:00:56,491 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676145653041 table=testCorruptedSnapshot-1676145653041 type=DISABLED ttl=0 } 2023-02-11 20:00:56,493 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:56,493 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:56,494 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:56,493 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:56,494 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:56,494 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:56,494 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:56,494 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:56,494 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:56,495 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:56,494 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:56,495 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:56,494 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 20:00:56,495 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 20:00:56,496 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf/726be4c9648c48bf9dccefde2083eb5e] hfiles 2023-02-11 20:00:56,496 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf/726be4c9648c48bf9dccefde2083eb5e 2023-02-11 20:00:56,497 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf/c59ef0e4ab8e4494b8187a90d196014d] hfiles 2023-02-11 20:00:56,497 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf/c59ef0e4ab8e4494b8187a90d196014d 2023-02-11 20:00:56,497 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf/583edb103c644b3182f51b355c9fe11d] hfiles 2023-02-11 20:00:56,497 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf/583edb103c644b3182f51b355c9fe11d 2023-02-11 20:00:56,497 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf/8f8d90d1d9fe4323a8458e43b284b07f] hfiles 2023-02-11 20:00:56,497 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf/8f8d90d1d9fe4323a8458e43b284b07f 2023-02-11 20:00:56,498 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf/f549005d553c4c269a0d1574c1fcb42b] hfiles 2023-02-11 20:00:56,498 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf/f549005d553c4c269a0d1574c1fcb42b 2023-02-11 20:00:56,498 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf/e0b7ecd250ec4bd99377381ecb2a6f2a] hfiles 2023-02-11 20:00:56,498 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf/e0b7ecd250ec4bd99377381ecb2a6f2a 2023-02-11 20:00:56,502 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/a87ff679a2f3e71d9181a67b7542122c20230211dcc9bcc5d07b49e88321a7e2f185f411, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c4ca4238a0b923820dcc509a6f75849b20230211bee74755269547a78094d6514813ce6e, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c81e728d9d4c2f636f067f89cc14862c20230211847278d687ef4e8094a419ca4751f0b4, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/d41d8cd98f00b204e9800998ecf8427e20230211a615ab38ba894a7bb51b2d27eb43d491, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/e4da3b7fbbce2345d7772b0674a318d5202302117a42971387dc4e4abe5be69050d0dbca, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211f8be42215c3f4b14923eab8a227b17a6] hfiles 2023-02-11 20:00:56,502 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/a87ff679a2f3e71d9181a67b7542122c20230211dcc9bcc5d07b49e88321a7e2f185f411 2023-02-11 20:00:56,502 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c4ca4238a0b923820dcc509a6f75849b20230211bee74755269547a78094d6514813ce6e 2023-02-11 20:00:56,502 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c81e728d9d4c2f636f067f89cc14862c20230211847278d687ef4e8094a419ca4751f0b4 2023-02-11 20:00:56,502 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/d41d8cd98f00b204e9800998ecf8427e20230211a615ab38ba894a7bb51b2d27eb43d491 2023-02-11 20:00:56,502 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/e4da3b7fbbce2345d7772b0674a318d5202302117a42971387dc4e4abe5be69050d0dbca 2023-02-11 20:00:56,502 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211f8be42215c3f4b14923eab8a227b17a6 2023-02-11 20:00:56,618 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676145653041 table=testCorruptedSnapshot-1676145653041 type=DISABLED ttl=0 } as finished. 2023-02-11 20:00:56,619 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676145653041 2023-02-11 20:00:56,619 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/snaptb0-1676145653041 2023-02-11 20:00:56,689 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/snaptb0-1676145653041 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/snaptb0-1676145653041 2023-02-11 20:00:56,715 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 20:00:56,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676145653041 table=testCorruptedSnapshot-1676145653041 type=FLUSH ttl=0 } is done 2023-02-11 20:00:56,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(404): Snapshoting '{ ss=snaptb0-1676145653041 table=testCorruptedSnapshot-1676145653041 type=FLUSH ttl=0 }' is still in progress! 2023-02-11 20:00:56,717 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-11 20:00:56,743 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676145653041 of table testCorruptedSnapshot-1676145653041 completed 2023-02-11 20:00:56,743 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/snaptb0-1676145653041 2023-02-11 20:00:56,744 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.hbase-snapshot/.tmp/snaptb0-1676145653041 2023-02-11 20:00:56,744 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676145653041 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676145653041 at 1676145656464Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676145653041 table=testCorruptedSnapshot-1676145653041 type=DISABLED ttl=0 } at 1676145656491 (+27 ms)Consolidate snapshot: snaptb0-1676145653041 at 1676145656619 (+128 ms)Loading Region manifests for snaptb0-1676145653041 at 1676145656619Writing data manifest for snaptb0-1676145653041 at 1676145656633 (+14 ms)Verifying snapshot: snaptb0-1676145653041 at 1676145656651 (+18 ms)Snapshot snaptb0-1676145653041 of table testCorruptedSnapshot-1676145653041 completed at 1676145656743 (+92 ms) 2023-02-11 20:00:56,745 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676145653041, type=EXCLUSIVE 2023-02-11 20:00:56,748 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676145653041, type=EXCLUSIVE in 291 msec 2023-02-11 20:00:57,217 DEBUG [Listener at localhost/33691] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 20:00:57,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676145653041 table=testCorruptedSnapshot-1676145653041 type=FLUSH ttl=0 } is done 2023-02-11 20:00:57,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676145653041 table=testCorruptedSnapshot-1676145653041 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 20:00:57,219 INFO [Listener at localhost/33691] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676145653041 2023-02-11 20:00:57,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676145653041 2023-02-11 20:00:57,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676145653041 2023-02-11 20:00:57,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-11 20:00:57,222 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676145653041 2023-02-11 20:00:57,223 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145657222"}]},"ts":"1676145657222"} 2023-02-11 20:00:57,224 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676145653041, state=ENABLING in hbase:meta 2023-02-11 20:00:57,226 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 20:00:57,226 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, ASSIGN}] 2023-02-11 20:00:57,227 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, ASSIGN 2023-02-11 20:00:57,227 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, ASSIGN 2023-02-11 20:00:57,227 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, ASSIGN 2023-02-11 20:00:57,227 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, ASSIGN 2023-02-11 20:00:57,227 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:57,227 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35937,1676145619887; forceNewPlan=false, retain=true 2023-02-11 20:00:57,227 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:57,228 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44175,1676145619924; forceNewPlan=false, retain=true 2023-02-11 20:00:57,228 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:57,228 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:57,228 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42011,1676145619841; forceNewPlan=false, retain=true 2023-02-11 20:00:57,228 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, ASSIGN 2023-02-11 20:00:57,228 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42011,1676145619841; forceNewPlan=false, retain=true 2023-02-11 20:00:57,228 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, ASSIGN 2023-02-11 20:00:57,229 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:57,229 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:57,229 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35937,1676145619887; forceNewPlan=false, retain=true 2023-02-11 20:00:57,229 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44175,1676145619924; forceNewPlan=false, retain=true 2023-02-11 20:00:57,378 INFO [jenkins-hbase4:39971] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 20:00:57,378 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=2f7f00f697d9b55aa03c57a3213d5a7c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:57,378 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=5bc4c474972588bf6c06dc75b6e4aea6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:57,378 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=3518e490e7f4ccb7c60b36f15d4f9183, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:57,378 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145657378"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145657378"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145657378"}]},"ts":"1676145657378"} 2023-02-11 20:00:57,378 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=38dcb256366159437c4db1abc8d1671c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:57,378 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=7c1f214213ec2e6ba08d051362b2ca4e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:57,378 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145657378"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145657378"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145657378"}]},"ts":"1676145657378"} 2023-02-11 20:00:57,378 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145657378"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145657378"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145657378"}]},"ts":"1676145657378"} 2023-02-11 20:00:57,378 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145657378"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145657378"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145657378"}]},"ts":"1676145657378"} 2023-02-11 20:00:57,378 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145657378"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145657378"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145657378"}]},"ts":"1676145657378"} 2023-02-11 20:00:57,379 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=257, state=RUNNABLE; OpenRegionProcedure 5bc4c474972588bf6c06dc75b6e4aea6, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:57,380 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=261, state=RUNNABLE; OpenRegionProcedure 38dcb256366159437c4db1abc8d1671c, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:57,381 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=258, state=RUNNABLE; OpenRegionProcedure 7c1f214213ec2e6ba08d051362b2ca4e, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:57,381 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=259, state=RUNNABLE; OpenRegionProcedure 3518e490e7f4ccb7c60b36f15d4f9183, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:57,382 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=260, state=RUNNABLE; OpenRegionProcedure 2f7f00f697d9b55aa03c57a3213d5a7c, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:57,384 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=3eae0a3c58bb2f53c1289450916a28ea, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:57,384 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145657384"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145657384"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145657384"}]},"ts":"1676145657384"} 2023-02-11 20:00:57,386 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=262, state=RUNNABLE; OpenRegionProcedure 3eae0a3c58bb2f53c1289450916a28ea, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:57,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-11 20:00:57,534 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:57,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2f7f00f697d9b55aa03c57a3213d5a7c, NAME => 'testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 20:00:57,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:57,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:57,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:57,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:57,534 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:57,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 38dcb256366159437c4db1abc8d1671c, NAME => 'testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 20:00:57,535 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:57,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7c1f214213ec2e6ba08d051362b2ca4e, NAME => 'testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 20:00:57,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:57,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:57,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:57,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:57,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:57,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:57,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:57,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:57,535 INFO [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:57,536 INFO [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:57,536 DEBUG [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf 2023-02-11 20:00:57,536 DEBUG [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf 2023-02-11 20:00:57,536 INFO [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:57,537 INFO [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2f7f00f697d9b55aa03c57a3213d5a7c columnFamilyName cf 2023-02-11 20:00:57,537 DEBUG [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf 2023-02-11 20:00:57,537 DEBUG [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf 2023-02-11 20:00:57,537 DEBUG [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf 2023-02-11 20:00:57,537 DEBUG [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf 2023-02-11 20:00:57,537 INFO [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 38dcb256366159437c4db1abc8d1671c columnFamilyName cf 2023-02-11 20:00:57,537 INFO [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7c1f214213ec2e6ba08d051362b2ca4e columnFamilyName cf 2023-02-11 20:00:57,540 DEBUG [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf/726be4c9648c48bf9dccefde2083eb5e 2023-02-11 20:00:57,540 INFO [StoreOpener-2f7f00f697d9b55aa03c57a3213d5a7c-1] regionserver.HStore(310): Store=2f7f00f697d9b55aa03c57a3213d5a7c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:57,541 DEBUG [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf/f549005d553c4c269a0d1574c1fcb42b 2023-02-11 20:00:57,541 DEBUG [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf/e0b7ecd250ec4bd99377381ecb2a6f2a 2023-02-11 20:00:57,541 INFO [StoreOpener-38dcb256366159437c4db1abc8d1671c-1] regionserver.HStore(310): Store=38dcb256366159437c4db1abc8d1671c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:57,541 INFO [StoreOpener-7c1f214213ec2e6ba08d051362b2ca4e-1] regionserver.HStore(310): Store=7c1f214213ec2e6ba08d051362b2ca4e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:57,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:57,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:57,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:57,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:57,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:57,543 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:57,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:57,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:57,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:57,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 38dcb256366159437c4db1abc8d1671c; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67249150, jitterRate=0.0020904242992401123}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:57,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2f7f00f697d9b55aa03c57a3213d5a7c; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70965198, jitterRate=0.05746385455131531}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:57,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 38dcb256366159437c4db1abc8d1671c: Running coprocessor pre-open hook at 1676145657535Writing region info on filesystem at 1676145657535Initializing all the Stores at 1676145657536 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145657536Cleaning up temporary data from old regions at 1676145657543 (+7 ms)Cleaning up detritus from prior splits at 1676145657543Running coprocessor post-open hooks at 1676145657546 (+3 ms)Region opened successfully at 1676145657546 2023-02-11 20:00:57,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2f7f00f697d9b55aa03c57a3213d5a7c: Running coprocessor pre-open hook at 1676145657534Writing region info on filesystem at 1676145657534Initializing all the Stores at 1676145657535 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145657535Cleaning up temporary data from old regions at 1676145657543 (+8 ms)Cleaning up detritus from prior splits at 1676145657543Running coprocessor post-open hooks at 1676145657546 (+3 ms)Region opened successfully at 1676145657546 2023-02-11 20:00:57,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7c1f214213ec2e6ba08d051362b2ca4e; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60348313, jitterRate=-0.10074006021022797}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:57,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7c1f214213ec2e6ba08d051362b2ca4e: Running coprocessor pre-open hook at 1676145657535Writing region info on filesystem at 1676145657535Initializing all the Stores at 1676145657536 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145657536Cleaning up temporary data from old regions at 1676145657543 (+7 ms)Cleaning up detritus from prior splits at 1676145657543Running coprocessor post-open hooks at 1676145657546 (+3 ms)Region opened successfully at 1676145657546 2023-02-11 20:00:57,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c., pid=267, masterSystemTime=1676145657532 2023-02-11 20:00:57,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c., pid=264, masterSystemTime=1676145657533 2023-02-11 20:00:57,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e., pid=265, masterSystemTime=1676145657533 2023-02-11 20:00:57,547 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:57,547 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:57,548 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:57,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5bc4c474972588bf6c06dc75b6e4aea6, NAME => 'testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 20:00:57,548 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=2f7f00f697d9b55aa03c57a3213d5a7c, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:57,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:57,548 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145657548"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145657548"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145657548"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145657548"}]},"ts":"1676145657548"} 2023-02-11 20:00:57,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:57,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:57,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:57,548 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:57,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:57,548 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:57,548 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=38dcb256366159437c4db1abc8d1671c, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:57,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3518e490e7f4ccb7c60b36f15d4f9183, NAME => 'testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 20:00:57,548 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145657548"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145657548"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145657548"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145657548"}]},"ts":"1676145657548"} 2023-02-11 20:00:57,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:57,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:57,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:57,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:57,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:57,549 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:57,549 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:57,549 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=7c1f214213ec2e6ba08d051362b2ca4e, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:57,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3eae0a3c58bb2f53c1289450916a28ea, NAME => 'testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 20:00:57,549 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145657549"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145657549"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145657549"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145657549"}]},"ts":"1676145657549"} 2023-02-11 20:00:57,549 INFO [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:57,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676145653041 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:57,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 20:00:57,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:57,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:57,550 INFO [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:57,550 DEBUG [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf 2023-02-11 20:00:57,550 DEBUG [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf 2023-02-11 20:00:57,551 INFO [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5bc4c474972588bf6c06dc75b6e4aea6 columnFamilyName cf 2023-02-11 20:00:57,551 DEBUG [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf 2023-02-11 20:00:57,551 DEBUG [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf 2023-02-11 20:00:57,551 INFO [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:57,551 INFO [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3518e490e7f4ccb7c60b36f15d4f9183 columnFamilyName cf 2023-02-11 20:00:57,552 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=260 2023-02-11 20:00:57,552 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=260, state=SUCCESS; OpenRegionProcedure 2f7f00f697d9b55aa03c57a3213d5a7c, server=jenkins-hbase4.apache.org,42011,1676145619841 in 167 msec 2023-02-11 20:00:57,552 DEBUG [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf 2023-02-11 20:00:57,552 DEBUG [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf 2023-02-11 20:00:57,553 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=261 2023-02-11 20:00:57,553 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=261, state=SUCCESS; OpenRegionProcedure 38dcb256366159437c4db1abc8d1671c, server=jenkins-hbase4.apache.org,44175,1676145619924 in 170 msec 2023-02-11 20:00:57,553 INFO [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3eae0a3c58bb2f53c1289450916a28ea columnFamilyName cf 2023-02-11 20:00:57,553 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, ASSIGN in 326 msec 2023-02-11 20:00:57,554 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, ASSIGN in 327 msec 2023-02-11 20:00:57,555 DEBUG [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf/8f8d90d1d9fe4323a8458e43b284b07f 2023-02-11 20:00:57,556 INFO [StoreOpener-3518e490e7f4ccb7c60b36f15d4f9183-1] regionserver.HStore(310): Store=3518e490e7f4ccb7c60b36f15d4f9183/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:57,556 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:57,557 DEBUG [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf/c59ef0e4ab8e4494b8187a90d196014d 2023-02-11 20:00:57,557 INFO [StoreOpener-3eae0a3c58bb2f53c1289450916a28ea-1] regionserver.HStore(310): Store=3eae0a3c58bb2f53c1289450916a28ea/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:57,557 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:57,557 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=258 2023-02-11 20:00:57,557 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=258, state=SUCCESS; OpenRegionProcedure 7c1f214213ec2e6ba08d051362b2ca4e, server=jenkins-hbase4.apache.org,35937,1676145619887 in 175 msec 2023-02-11 20:00:57,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:57,558 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, ASSIGN in 331 msec 2023-02-11 20:00:57,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:57,559 DEBUG [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] regionserver.HStore(539): loaded hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf/583edb103c644b3182f51b355c9fe11d 2023-02-11 20:00:57,559 INFO [StoreOpener-5bc4c474972588bf6c06dc75b6e4aea6-1] regionserver.HStore(310): Store=5bc4c474972588bf6c06dc75b6e4aea6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 20:00:57,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:57,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:57,561 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3518e490e7f4ccb7c60b36f15d4f9183; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70273168, jitterRate=0.047151803970336914}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:57,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3518e490e7f4ccb7c60b36f15d4f9183: Running coprocessor pre-open hook at 1676145657549Writing region info on filesystem at 1676145657549Initializing all the Stores at 1676145657550 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145657550Cleaning up temporary data from old regions at 1676145657558 (+8 ms)Cleaning up detritus from prior splits at 1676145657558Running coprocessor post-open hooks at 1676145657561 (+3 ms)Region opened successfully at 1676145657561 2023-02-11 20:00:57,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:57,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:57,561 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183., pid=266, masterSystemTime=1676145657533 2023-02-11 20:00:57,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3eae0a3c58bb2f53c1289450916a28ea; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72486277, jitterRate=0.0801296979188919}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:57,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3eae0a3c58bb2f53c1289450916a28ea: Running coprocessor pre-open hook at 1676145657550Writing region info on filesystem at 1676145657550Initializing all the Stores at 1676145657551 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145657551Cleaning up temporary data from old regions at 1676145657559 (+8 ms)Cleaning up detritus from prior splits at 1676145657559Running coprocessor post-open hooks at 1676145657562 (+3 ms)Region opened successfully at 1676145657562 2023-02-11 20:00:57,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:57,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:57,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea., pid=268, masterSystemTime=1676145657533 2023-02-11 20:00:57,563 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=3518e490e7f4ccb7c60b36f15d4f9183, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:57,563 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145657563"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145657563"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145657563"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145657563"}]},"ts":"1676145657563"} 2023-02-11 20:00:57,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:57,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:57,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:57,564 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=3eae0a3c58bb2f53c1289450916a28ea, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:57,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5bc4c474972588bf6c06dc75b6e4aea6; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67270442, jitterRate=0.0024076998233795166}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 20:00:57,564 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145657564"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145657564"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145657564"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145657564"}]},"ts":"1676145657564"} 2023-02-11 20:00:57,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5bc4c474972588bf6c06dc75b6e4aea6: Running coprocessor pre-open hook at 1676145657548Writing region info on filesystem at 1676145657548Initializing all the Stores at 1676145657549 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676145657549Cleaning up temporary data from old regions at 1676145657561 (+12 ms)Cleaning up detritus from prior splits at 1676145657561Running coprocessor post-open hooks at 1676145657564 (+3 ms)Region opened successfully at 1676145657564 2023-02-11 20:00:57,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6., pid=263, masterSystemTime=1676145657532 2023-02-11 20:00:57,565 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=259 2023-02-11 20:00:57,565 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=259, state=SUCCESS; OpenRegionProcedure 3518e490e7f4ccb7c60b36f15d4f9183, server=jenkins-hbase4.apache.org,44175,1676145619924 in 183 msec 2023-02-11 20:00:57,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:57,566 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:57,566 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=5bc4c474972588bf6c06dc75b6e4aea6, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:57,566 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145657566"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676145657566"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676145657566"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676145657566"}]},"ts":"1676145657566"} 2023-02-11 20:00:57,566 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=262 2023-02-11 20:00:57,566 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, ASSIGN in 339 msec 2023-02-11 20:00:57,566 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=262, state=SUCCESS; OpenRegionProcedure 3eae0a3c58bb2f53c1289450916a28ea, server=jenkins-hbase4.apache.org,35937,1676145619887 in 179 msec 2023-02-11 20:00:57,567 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, ASSIGN in 340 msec 2023-02-11 20:00:57,568 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=257 2023-02-11 20:00:57,568 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=257, state=SUCCESS; OpenRegionProcedure 5bc4c474972588bf6c06dc75b6e4aea6, server=jenkins-hbase4.apache.org,42011,1676145619841 in 188 msec 2023-02-11 20:00:57,569 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=257, resume processing ppid=256 2023-02-11 20:00:57,569 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, ASSIGN in 342 msec 2023-02-11 20:00:57,569 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145657569"}]},"ts":"1676145657569"} 2023-02-11 20:00:57,570 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676145653041, state=ENABLED in hbase:meta 2023-02-11 20:00:57,572 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676145653041' was successfully enabled. 2023-02-11 20:00:57,573 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676145653041 in 353 msec 2023-02-11 20:00:57,646 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 20:00:57,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-11 20:00:57,974 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676145653041, procId: 256 completed 2023-02-11 20:00:57,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:57,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44175] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:57,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:57,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42011] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:57,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:57,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35937] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 20:00:57,994 DEBUG [Listener at localhost/33691] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676145653041 2023-02-11 20:00:57,995 DEBUG [Listener at localhost/33691] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:57,995 INFO [Listener at localhost/33691] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 20:00:58,076 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69705b98 to 127.0.0.1:51732 2023-02-11 20:00:58,076 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:00:58,179 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 20:00:58,287 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 20:00:58,398 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 20:00:58,529 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 20:00:58,642 DEBUG [Listener at localhost/33691] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 20:00:59,432 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: 726be4c9648c48bf9dccefde2083eb5e in the real (hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf/726be4c9648c48bf9dccefde2083eb5e) or archive (hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf/726be4c9648c48bf9dccefde2083eb5e) directory for the primary table. 2023-02-11 20:00:59,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 726be4c9648c48bf9dccefde2083eb5e in the real (hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf/726be4c9648c48bf9dccefde2083eb5e) or archive (hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf/726be4c9648c48bf9dccefde2083eb5e) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2493) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2489) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 20:00:59,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] ipc.CallRunner(144): callId: 372 service: MasterService methodName: RestoreSnapshot size: 116 connection: 172.31.14.131:39862 deadline: 1676145719428, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 726be4c9648c48bf9dccefde2083eb5e in the real (hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf/726be4c9648c48bf9dccefde2083eb5e) or archive (hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf/726be4c9648c48bf9dccefde2083eb5e) directory for the primary table. 2023-02-11 20:00:59,452 INFO [Listener at localhost/33691] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676145653041 2023-02-11 20:00:59,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676145653041 2023-02-11 20:00:59,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676145653041 2023-02-11 20:00:59,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-11 20:00:59,455 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145659455"}]},"ts":"1676145659455"} 2023-02-11 20:00:59,456 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676145653041, state=DISABLING in hbase:meta 2023-02-11 20:00:59,459 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676145653041 to state=DISABLING 2023-02-11 20:00:59,460 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, UNASSIGN}] 2023-02-11 20:00:59,461 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, UNASSIGN 2023-02-11 20:00:59,461 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, UNASSIGN 2023-02-11 20:00:59,461 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, UNASSIGN 2023-02-11 20:00:59,461 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, UNASSIGN 2023-02-11 20:00:59,461 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, UNASSIGN 2023-02-11 20:00:59,461 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=3518e490e7f4ccb7c60b36f15d4f9183, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:59,461 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145659461"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145659461"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145659461"}]},"ts":"1676145659461"} 2023-02-11 20:00:59,461 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=5bc4c474972588bf6c06dc75b6e4aea6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:59,461 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=7c1f214213ec2e6ba08d051362b2ca4e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:59,461 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145659461"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145659461"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145659461"}]},"ts":"1676145659461"} 2023-02-11 20:00:59,462 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=2f7f00f697d9b55aa03c57a3213d5a7c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:00:59,462 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145659461"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145659461"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145659461"}]},"ts":"1676145659461"} 2023-02-11 20:00:59,462 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=3eae0a3c58bb2f53c1289450916a28ea, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:00:59,462 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145659461"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145659461"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145659461"}]},"ts":"1676145659461"} 2023-02-11 20:00:59,462 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145659462"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145659462"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145659462"}]},"ts":"1676145659462"} 2023-02-11 20:00:59,462 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=273, state=RUNNABLE; CloseRegionProcedure 3518e490e7f4ccb7c60b36f15d4f9183, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:59,463 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=271, state=RUNNABLE; CloseRegionProcedure 5bc4c474972588bf6c06dc75b6e4aea6, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:59,463 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=272, state=RUNNABLE; CloseRegionProcedure 7c1f214213ec2e6ba08d051362b2ca4e, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:59,464 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=274, state=RUNNABLE; CloseRegionProcedure 2f7f00f697d9b55aa03c57a3213d5a7c, server=jenkins-hbase4.apache.org,42011,1676145619841}] 2023-02-11 20:00:59,465 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=276, state=RUNNABLE; CloseRegionProcedure 3eae0a3c58bb2f53c1289450916a28ea, server=jenkins-hbase4.apache.org,35937,1676145619887}] 2023-02-11 20:00:59,465 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, UNASSIGN 2023-02-11 20:00:59,466 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=38dcb256366159437c4db1abc8d1671c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:00:59,466 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145659466"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676145659466"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676145659466"}]},"ts":"1676145659466"} 2023-02-11 20:00:59,467 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=275, state=RUNNABLE; CloseRegionProcedure 38dcb256366159437c4db1abc8d1671c, server=jenkins-hbase4.apache.org,44175,1676145619924}] 2023-02-11 20:00:59,613 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:59,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3518e490e7f4ccb7c60b36f15d4f9183, disabling compactions & flushes 2023-02-11 20:00:59,613 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:59,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:59,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. after waiting 0 ms 2023-02-11 20:00:59,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:59,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 3518e490e7f4ccb7c60b36f15d4f9183 1/1 column families, dataSize=1.57 KB heapSize=3.63 KB 2023-02-11 20:00:59,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:59,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2f7f00f697d9b55aa03c57a3213d5a7c, disabling compactions & flushes 2023-02-11 20:00:59,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:59,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:59,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. after waiting 0 ms 2023-02-11 20:00:59,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:59,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 2f7f00f697d9b55aa03c57a3213d5a7c 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-11 20:00:59,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:59,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3eae0a3c58bb2f53c1289450916a28ea, disabling compactions & flushes 2023-02-11 20:00:59,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:59,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:59,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. after waiting 0 ms 2023-02-11 20:00:59,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:59,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 3eae0a3c58bb2f53c1289450916a28ea 1/1 column families, dataSize=22.11 KB heapSize=47.78 KB 2023-02-11 20:00:59,649 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230211fdd423b90d6c40fd8ddeeec265630199 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c81e728d9d4c2f636f067f89cc14862c20230211fdd423b90d6c40fd8ddeeec265630199 2023-02-11 20:00:59,651 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d52023021143585c2bfb1e4f68b76e5ee7d3ce2a13 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/e4da3b7fbbce2345d7772b0674a318d52023021143585c2bfb1e4f68b76e5ee7d3ce2a13 2023-02-11 20:00:59,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230211a994002e11b141b282516e1e2d1e595d to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211a994002e11b141b282516e1e2d1e595d 2023-02-11 20:00:59,664 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/.tmp/cf/4be907d67230422090dc2690372ebeb6 2023-02-11 20:00:59,664 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/.tmp/cf/5d13d1bec25c4191ad9eb43760a680ab 2023-02-11 20:00:59,667 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/.tmp/cf/ddf71631d972477ab02979271c549c2f 2023-02-11 20:00:59,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/.tmp/cf/4be907d67230422090dc2690372ebeb6 as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf/4be907d67230422090dc2690372ebeb6 2023-02-11 20:00:59,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/.tmp/cf/5d13d1bec25c4191ad9eb43760a680ab as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf/5d13d1bec25c4191ad9eb43760a680ab 2023-02-11 20:00:59,671 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/.tmp/cf/ddf71631d972477ab02979271c549c2f as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf/ddf71631d972477ab02979271c549c2f 2023-02-11 20:00:59,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf/5d13d1bec25c4191ad9eb43760a680ab, entries=24, sequenceid=15, filesize=9.0 K 2023-02-11 20:00:59,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf/4be907d67230422090dc2690372ebeb6, entries=39, sequenceid=15, filesize=11.6 K 2023-02-11 20:00:59,675 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.57 KB/1606, heapSize ~3.61 KB/3696, currentSize=0 B/0 for 3518e490e7f4ccb7c60b36f15d4f9183 in 59ms, sequenceid=15, compaction requested=false 2023-02-11 20:00:59,675 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 2f7f00f697d9b55aa03c57a3213d5a7c in 59ms, sequenceid=15, compaction requested=false 2023-02-11 20:00:59,676 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf/ddf71631d972477ab02979271c549c2f, entries=338, sequenceid=15, filesize=63.2 K 2023-02-11 20:00:59,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.11 KB/22642, heapSize ~47.77 KB/48912, currentSize=0 B/0 for 3eae0a3c58bb2f53c1289450916a28ea in 60ms, sequenceid=15, compaction requested=false 2023-02-11 20:00:59,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 20:00:59,679 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 20:00:59,680 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. 2023-02-11 20:00:59,680 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. 2023-02-11 20:00:59,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2f7f00f697d9b55aa03c57a3213d5a7c: Waiting for close lock at 1676145659616Running coprocessor pre-close hooks at 1676145659616Disabling compacts and flushes for region at 1676145659616Disabling writes for close at 1676145659616Obtaining lock to block concurrent updates at 1676145659616Preparing flush snapshotting stores in 2f7f00f697d9b55aa03c57a3213d5a7c at 1676145659616Finished memstore snapshotting testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c., syncing WAL and waiting on mvcc, flushsize=dataSize=2611, getHeapSize=5856, getOffHeapSize=0, getCellsCount=39 at 1676145659616Flushing stores of testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c. at 1676145659617 (+1 ms)Flushing 2f7f00f697d9b55aa03c57a3213d5a7c/cf: creating writer at 1676145659618 (+1 ms)Flushing mob file 2f7f00f697d9b55aa03c57a3213d5a7c/cf: appending metadata at 1676145659623 (+5 ms)Flushing mob file 2f7f00f697d9b55aa03c57a3213d5a7c/cf: closing flushed file at 1676145659623Flushing 2f7f00f697d9b55aa03c57a3213d5a7c/cf: appending metadata at 1676145659654 (+31 ms)Flushing 2f7f00f697d9b55aa03c57a3213d5a7c/cf: closing flushed file at 1676145659654Flushing 2f7f00f697d9b55aa03c57a3213d5a7c/cf: reopening flushed file at 1676145659669 (+15 ms)Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 2f7f00f697d9b55aa03c57a3213d5a7c in 59ms, sequenceid=15, compaction requested=false at 1676145659675 (+6 ms)Writing region close event to WAL at 1676145659676 (+1 ms)Running coprocessor post-close hooks at 1676145659679 (+3 ms)Closed at 1676145659680 (+1 ms) 2023-02-11 20:00:59,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3518e490e7f4ccb7c60b36f15d4f9183: Waiting for close lock at 1676145659613Running coprocessor pre-close hooks at 1676145659613Disabling compacts and flushes for region at 1676145659613Disabling writes for close at 1676145659614 (+1 ms)Obtaining lock to block concurrent updates at 1676145659614Preparing flush snapshotting stores in 3518e490e7f4ccb7c60b36f15d4f9183 at 1676145659614Finished memstore snapshotting testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183., syncing WAL and waiting on mvcc, flushsize=dataSize=1606, getHeapSize=3696, getOffHeapSize=0, getCellsCount=24 at 1676145659614Flushing stores of testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183. at 1676145659614Flushing 3518e490e7f4ccb7c60b36f15d4f9183/cf: creating writer at 1676145659616 (+2 ms)Flushing mob file 3518e490e7f4ccb7c60b36f15d4f9183/cf: appending metadata at 1676145659621 (+5 ms)Flushing mob file 3518e490e7f4ccb7c60b36f15d4f9183/cf: closing flushed file at 1676145659621Flushing 3518e490e7f4ccb7c60b36f15d4f9183/cf: appending metadata at 1676145659650 (+29 ms)Flushing 3518e490e7f4ccb7c60b36f15d4f9183/cf: closing flushed file at 1676145659650Flushing 3518e490e7f4ccb7c60b36f15d4f9183/cf: reopening flushed file at 1676145659669 (+19 ms)Finished flush of dataSize ~1.57 KB/1606, heapSize ~3.61 KB/3696, currentSize=0 B/0 for 3518e490e7f4ccb7c60b36f15d4f9183 in 59ms, sequenceid=15, compaction requested=false at 1676145659675 (+6 ms)Writing region close event to WAL at 1676145659676 (+1 ms)Running coprocessor post-close hooks at 1676145659680 (+4 ms)Closed at 1676145659680 2023-02-11 20:00:59,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 20:00:59,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:00:59,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. 2023-02-11 20:00:59,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:59,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3eae0a3c58bb2f53c1289450916a28ea: Waiting for close lock at 1676145659616Running coprocessor pre-close hooks at 1676145659616Disabling compacts and flushes for region at 1676145659616Disabling writes for close at 1676145659616Obtaining lock to block concurrent updates at 1676145659617 (+1 ms)Preparing flush snapshotting stores in 3eae0a3c58bb2f53c1289450916a28ea at 1676145659617Finished memstore snapshotting testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea., syncing WAL and waiting on mvcc, flushsize=dataSize=22642, getHeapSize=48912, getOffHeapSize=0, getCellsCount=338 at 1676145659617Flushing stores of testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea. at 1676145659617Flushing 3eae0a3c58bb2f53c1289450916a28ea/cf: creating writer at 1676145659618 (+1 ms)Flushing mob file 3eae0a3c58bb2f53c1289450916a28ea/cf: appending metadata at 1676145659625 (+7 ms)Flushing mob file 3eae0a3c58bb2f53c1289450916a28ea/cf: closing flushed file at 1676145659625Flushing 3eae0a3c58bb2f53c1289450916a28ea/cf: appending metadata at 1676145659653 (+28 ms)Flushing 3eae0a3c58bb2f53c1289450916a28ea/cf: closing flushed file at 1676145659653Flushing 3eae0a3c58bb2f53c1289450916a28ea/cf: reopening flushed file at 1676145659671 (+18 ms)Finished flush of dataSize ~22.11 KB/22642, heapSize ~47.77 KB/48912, currentSize=0 B/0 for 3eae0a3c58bb2f53c1289450916a28ea in 60ms, sequenceid=15, compaction requested=false at 1676145659677 (+6 ms)Writing region close event to WAL at 1676145659678 (+1 ms)Running coprocessor post-close hooks at 1676145659682 (+4 ms)Closed at 1676145659683 (+1 ms) 2023-02-11 20:00:59,683 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=2f7f00f697d9b55aa03c57a3213d5a7c, regionState=CLOSED 2023-02-11 20:00:59,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5bc4c474972588bf6c06dc75b6e4aea6, disabling compactions & flushes 2023-02-11 20:00:59,683 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145659683"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145659683"}]},"ts":"1676145659683"} 2023-02-11 20:00:59,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:59,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:00:59,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:59,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:59,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. after waiting 0 ms 2023-02-11 20:00:59,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:59,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 38dcb256366159437c4db1abc8d1671c, disabling compactions & flushes 2023-02-11 20:00:59,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 5bc4c474972588bf6c06dc75b6e4aea6 1/1 column families, dataSize=2.94 KB heapSize=6.58 KB 2023-02-11 20:00:59,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:59,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:59,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. after waiting 0 ms 2023-02-11 20:00:59,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:59,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 38dcb256366159437c4db1abc8d1671c 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-11 20:00:59,684 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=3518e490e7f4ccb7c60b36f15d4f9183, regionState=CLOSED 2023-02-11 20:00:59,684 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145659684"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145659684"}]},"ts":"1676145659684"} 2023-02-11 20:00:59,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:00:59,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:59,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7c1f214213ec2e6ba08d051362b2ca4e, disabling compactions & flushes 2023-02-11 20:00:59,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:59,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:59,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. after waiting 0 ms 2023-02-11 20:00:59,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:59,685 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=3eae0a3c58bb2f53c1289450916a28ea, regionState=CLOSED 2023-02-11 20:00:59,685 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145659685"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145659685"}]},"ts":"1676145659685"} 2023-02-11 20:00:59,686 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7c1f214213ec2e6ba08d051362b2ca4e 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-11 20:00:59,688 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=274 2023-02-11 20:00:59,688 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=274, state=SUCCESS; CloseRegionProcedure 2f7f00f697d9b55aa03c57a3213d5a7c, server=jenkins-hbase4.apache.org,42011,1676145619841 in 221 msec 2023-02-11 20:00:59,689 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=273 2023-02-11 20:00:59,689 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=273, state=SUCCESS; CloseRegionProcedure 3518e490e7f4ccb7c60b36f15d4f9183, server=jenkins-hbase4.apache.org,44175,1676145619924 in 224 msec 2023-02-11 20:00:59,690 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=276 2023-02-11 20:00:59,690 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=276, state=SUCCESS; CloseRegionProcedure 3eae0a3c58bb2f53c1289450916a28ea, server=jenkins-hbase4.apache.org,35937,1676145619887 in 222 msec 2023-02-11 20:00:59,690 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=2f7f00f697d9b55aa03c57a3213d5a7c, UNASSIGN in 228 msec 2023-02-11 20:00:59,690 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3518e490e7f4ccb7c60b36f15d4f9183, UNASSIGN in 229 msec 2023-02-11 20:00:59,691 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=3eae0a3c58bb2f53c1289450916a28ea, UNASSIGN in 230 msec 2023-02-11 20:00:59,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-11 20:00:59,708 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230211c867954c213d408496a0c00392a91008 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/a87ff679a2f3e71d9181a67b7542122c20230211c867954c213d408496a0c00392a91008 2023-02-11 20:00:59,710 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230211db6be44d53a94924b821944cd1bc0c82 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/d41d8cd98f00b204e9800998ecf8427e20230211db6be44d53a94924b821944cd1bc0c82 2023-02-11 20:00:59,713 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230211689f735cdb02427f88e276320159afad to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c4ca4238a0b923820dcc509a6f75849b20230211689f735cdb02427f88e276320159afad 2023-02-11 20:00:59,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/.tmp/cf/fe5bcf26597d423bbfb0c7a76a007ad9 2023-02-11 20:00:59,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/.tmp/cf/fe5bcf26597d423bbfb0c7a76a007ad9 as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf/fe5bcf26597d423bbfb0c7a76a007ad9 2023-02-11 20:00:59,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/.tmp/cf/bd26dae8c1a347f598c31c1c9a8110c8 2023-02-11 20:00:59,723 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf/fe5bcf26597d423bbfb0c7a76a007ad9, entries=27, sequenceid=15, filesize=9.6 K 2023-02-11 20:00:59,723 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/.tmp/cf/bd26dae8c1a347f598c31c1c9a8110c8 as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf/bd26dae8c1a347f598c31c1c9a8110c8 2023-02-11 20:00:59,723 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 38dcb256366159437c4db1abc8d1671c in 40ms, sequenceid=15, compaction requested=false 2023-02-11 20:00:59,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/.tmp/cf/a234d637657449b2b52b2d6d7c111d15 2023-02-11 20:00:59,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf/bd26dae8c1a347f598c31c1c9a8110c8, entries=45, sequenceid=15, filesize=12.6 K 2023-02-11 20:00:59,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 20:00:59,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/.tmp/cf/a234d637657449b2b52b2d6d7c111d15 as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf/a234d637657449b2b52b2d6d7c111d15 2023-02-11 20:00:59,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. 2023-02-11 20:00:59,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.94 KB/3013, heapSize ~6.56 KB/6720, currentSize=0 B/0 for 5bc4c474972588bf6c06dc75b6e4aea6 in 45ms, sequenceid=15, compaction requested=false 2023-02-11 20:00:59,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 38dcb256366159437c4db1abc8d1671c: Waiting for close lock at 1676145659683Running coprocessor pre-close hooks at 1676145659683Disabling compacts and flushes for region at 1676145659683Disabling writes for close at 1676145659683Obtaining lock to block concurrent updates at 1676145659684 (+1 ms)Preparing flush snapshotting stores in 38dcb256366159437c4db1abc8d1671c at 1676145659684Finished memstore snapshotting testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676145659684Flushing stores of testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c. at 1676145659684Flushing 38dcb256366159437c4db1abc8d1671c/cf: creating writer at 1676145659685 (+1 ms)Flushing mob file 38dcb256366159437c4db1abc8d1671c/cf: appending metadata at 1676145659690 (+5 ms)Flushing mob file 38dcb256366159437c4db1abc8d1671c/cf: closing flushed file at 1676145659690Flushing 38dcb256366159437c4db1abc8d1671c/cf: appending metadata at 1676145659709 (+19 ms)Flushing 38dcb256366159437c4db1abc8d1671c/cf: closing flushed file at 1676145659709Flushing 38dcb256366159437c4db1abc8d1671c/cf: reopening flushed file at 1676145659719 (+10 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 38dcb256366159437c4db1abc8d1671c in 40ms, sequenceid=15, compaction requested=false at 1676145659724 (+5 ms)Writing region close event to WAL at 1676145659724Running coprocessor post-close hooks at 1676145659727 (+3 ms)Closed at 1676145659728 (+1 ms) 2023-02-11 20:00:59,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 38dcb256366159437c4db1abc8d1671c 2023-02-11 20:00:59,730 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=38dcb256366159437c4db1abc8d1671c, regionState=CLOSED 2023-02-11 20:00:59,730 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145659730"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145659730"}]},"ts":"1676145659730"} 2023-02-11 20:00:59,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 20:00:59,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf/a234d637657449b2b52b2d6d7c111d15, entries=27, sequenceid=15, filesize=9.6 K 2023-02-11 20:00:59,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. 2023-02-11 20:00:59,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5bc4c474972588bf6c06dc75b6e4aea6: Waiting for close lock at 1676145659683Running coprocessor pre-close hooks at 1676145659683Disabling compacts and flushes for region at 1676145659683Disabling writes for close at 1676145659683Obtaining lock to block concurrent updates at 1676145659683Preparing flush snapshotting stores in 5bc4c474972588bf6c06dc75b6e4aea6 at 1676145659683Finished memstore snapshotting testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6., syncing WAL and waiting on mvcc, flushsize=dataSize=3013, getHeapSize=6720, getOffHeapSize=0, getCellsCount=45 at 1676145659684 (+1 ms)Flushing stores of testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6. at 1676145659684Flushing 5bc4c474972588bf6c06dc75b6e4aea6/cf: creating writer at 1676145659685 (+1 ms)Flushing mob file 5bc4c474972588bf6c06dc75b6e4aea6/cf: appending metadata at 1676145659690 (+5 ms)Flushing mob file 5bc4c474972588bf6c06dc75b6e4aea6/cf: closing flushed file at 1676145659690Flushing 5bc4c474972588bf6c06dc75b6e4aea6/cf: appending metadata at 1676145659711 (+21 ms)Flushing 5bc4c474972588bf6c06dc75b6e4aea6/cf: closing flushed file at 1676145659711Flushing 5bc4c474972588bf6c06dc75b6e4aea6/cf: reopening flushed file at 1676145659724 (+13 ms)Finished flush of dataSize ~2.94 KB/3013, heapSize ~6.56 KB/6720, currentSize=0 B/0 for 5bc4c474972588bf6c06dc75b6e4aea6 in 45ms, sequenceid=15, compaction requested=false at 1676145659728 (+4 ms)Writing region close event to WAL at 1676145659729 (+1 ms)Running coprocessor post-close hooks at 1676145659731 (+2 ms)Closed at 1676145659732 (+1 ms) 2023-02-11 20:00:59,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 7c1f214213ec2e6ba08d051362b2ca4e in 47ms, sequenceid=15, compaction requested=false 2023-02-11 20:00:59,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=275 2023-02-11 20:00:59,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=275, state=SUCCESS; CloseRegionProcedure 38dcb256366159437c4db1abc8d1671c, server=jenkins-hbase4.apache.org,44175,1676145619924 in 264 msec 2023-02-11 20:00:59,733 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:00:59,733 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=5bc4c474972588bf6c06dc75b6e4aea6, regionState=CLOSED 2023-02-11 20:00:59,734 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676145659733"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145659733"}]},"ts":"1676145659733"} 2023-02-11 20:00:59,734 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=38dcb256366159437c4db1abc8d1671c, UNASSIGN in 273 msec 2023-02-11 20:00:59,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 20:00:59,735 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=271 2023-02-11 20:00:59,736 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=271, state=SUCCESS; CloseRegionProcedure 5bc4c474972588bf6c06dc75b6e4aea6, server=jenkins-hbase4.apache.org,42011,1676145619841 in 271 msec 2023-02-11 20:00:59,736 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. 2023-02-11 20:00:59,736 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7c1f214213ec2e6ba08d051362b2ca4e: Waiting for close lock at 1676145659684Running coprocessor pre-close hooks at 1676145659684Disabling compacts and flushes for region at 1676145659684Disabling writes for close at 1676145659685 (+1 ms)Obtaining lock to block concurrent updates at 1676145659686 (+1 ms)Preparing flush snapshotting stores in 7c1f214213ec2e6ba08d051362b2ca4e at 1676145659686Finished memstore snapshotting testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676145659686Flushing stores of testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e. at 1676145659686Flushing 7c1f214213ec2e6ba08d051362b2ca4e/cf: creating writer at 1676145659687 (+1 ms)Flushing mob file 7c1f214213ec2e6ba08d051362b2ca4e/cf: appending metadata at 1676145659693 (+6 ms)Flushing mob file 7c1f214213ec2e6ba08d051362b2ca4e/cf: closing flushed file at 1676145659693Flushing 7c1f214213ec2e6ba08d051362b2ca4e/cf: appending metadata at 1676145659714 (+21 ms)Flushing 7c1f214213ec2e6ba08d051362b2ca4e/cf: closing flushed file at 1676145659714Flushing 7c1f214213ec2e6ba08d051362b2ca4e/cf: reopening flushed file at 1676145659728 (+14 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 7c1f214213ec2e6ba08d051362b2ca4e in 47ms, sequenceid=15, compaction requested=false at 1676145659732 (+4 ms)Writing region close event to WAL at 1676145659733 (+1 ms)Running coprocessor post-close hooks at 1676145659736 (+3 ms)Closed at 1676145659736 2023-02-11 20:00:59,737 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=5bc4c474972588bf6c06dc75b6e4aea6, UNASSIGN in 275 msec 2023-02-11 20:00:59,738 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:00:59,738 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=7c1f214213ec2e6ba08d051362b2ca4e, regionState=CLOSED 2023-02-11 20:00:59,738 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676145659738"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676145659738"}]},"ts":"1676145659738"} 2023-02-11 20:00:59,740 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=272 2023-02-11 20:00:59,740 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=272, state=SUCCESS; CloseRegionProcedure 7c1f214213ec2e6ba08d051362b2ca4e, server=jenkins-hbase4.apache.org,35937,1676145619887 in 276 msec 2023-02-11 20:00:59,741 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=272, resume processing ppid=270 2023-02-11 20:00:59,741 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676145653041, region=7c1f214213ec2e6ba08d051362b2ca4e, UNASSIGN in 280 msec 2023-02-11 20:00:59,741 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676145659741"}]},"ts":"1676145659741"} 2023-02-11 20:00:59,742 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676145653041, state=DISABLED in hbase:meta 2023-02-11 20:00:59,744 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676145653041 to state=DISABLED 2023-02-11 20:00:59,745 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676145653041 in 291 msec 2023-02-11 20:01:00,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-11 20:01:00,208 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676145653041, procId: 270 completed 2023-02-11 20:01:00,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testCorruptedSnapshot-1676145653041 2023-02-11 20:01:00,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1676145653041 2023-02-11 20:01:00,210 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676145653041 2023-02-11 20:01:00,211 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676145653041 2023-02-11 20:01:00,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-11 20:01:00,218 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:01:00,218 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:01:00,218 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c 2023-02-11 20:01:00,218 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:01:00,218 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:01:00,218 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:01:00,220 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/recovered.edits] 2023-02-11 20:01:00,220 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/recovered.edits] 2023-02-11 20:01:00,220 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/recovered.edits] 2023-02-11 20:01:00,221 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/recovered.edits] 2023-02-11 20:01:00,221 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/recovered.edits] 2023-02-11 20:01:00,221 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf, FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/recovered.edits] 2023-02-11 20:01:00,226 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf/f549005d553c4c269a0d1574c1fcb42b to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf/f549005d553c4c269a0d1574c1fcb42b 2023-02-11 20:01:00,226 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf/583edb103c644b3182f51b355c9fe11d to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf/583edb103c644b3182f51b355c9fe11d 2023-02-11 20:01:00,226 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf/4be907d67230422090dc2690372ebeb6 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/cf/4be907d67230422090dc2690372ebeb6 2023-02-11 20:01:00,226 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf/a234d637657449b2b52b2d6d7c111d15 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf/a234d637657449b2b52b2d6d7c111d15 2023-02-11 20:01:00,228 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf/c59ef0e4ab8e4494b8187a90d196014d to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf/c59ef0e4ab8e4494b8187a90d196014d 2023-02-11 20:01:00,229 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf/fe5bcf26597d423bbfb0c7a76a007ad9 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/cf/fe5bcf26597d423bbfb0c7a76a007ad9 2023-02-11 20:01:00,229 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf/5d13d1bec25c4191ad9eb43760a680ab to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf/5d13d1bec25c4191ad9eb43760a680ab 2023-02-11 20:01:00,229 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf/bd26dae8c1a347f598c31c1c9a8110c8 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/cf/bd26dae8c1a347f598c31c1c9a8110c8 2023-02-11 20:01:00,230 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf/e0b7ecd250ec4bd99377381ecb2a6f2a to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/cf/e0b7ecd250ec4bd99377381ecb2a6f2a 2023-02-11 20:01:00,230 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf/ddf71631d972477ab02979271c549c2f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/cf/ddf71631d972477ab02979271c549c2f 2023-02-11 20:01:00,233 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf/8f8d90d1d9fe4323a8458e43b284b07f to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/cf/8f8d90d1d9fe4323a8458e43b284b07f 2023-02-11 20:01:00,233 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/recovered.edits/18.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c/recovered.edits/18.seqid 2023-02-11 20:01:00,234 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/2f7f00f697d9b55aa03c57a3213d5a7c 2023-02-11 20:01:00,235 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/recovered.edits/18.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c/recovered.edits/18.seqid 2023-02-11 20:01:00,235 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/recovered.edits/18.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6/recovered.edits/18.seqid 2023-02-11 20:01:00,235 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/recovered.edits/18.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e/recovered.edits/18.seqid 2023-02-11 20:01:00,236 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/38dcb256366159437c4db1abc8d1671c 2023-02-11 20:01:00,236 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/5bc4c474972588bf6c06dc75b6e4aea6 2023-02-11 20:01:00,236 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/7c1f214213ec2e6ba08d051362b2ca4e 2023-02-11 20:01:00,238 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/recovered.edits/18.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183/recovered.edits/18.seqid 2023-02-11 20:01:00,238 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/recovered.edits/18.seqid to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea/recovered.edits/18.seqid 2023-02-11 20:01:00,238 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3518e490e7f4ccb7c60b36f15d4f9183 2023-02-11 20:01:00,238 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/.tmp/data/default/testCorruptedSnapshot-1676145653041/3eae0a3c58bb2f53c1289450916a28ea 2023-02-11 20:01:00,238 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676145653041 regions 2023-02-11 20:01:00,239 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368 2023-02-11 20:01:00,239 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf] 2023-02-11 20:01:00,242 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/a87ff679a2f3e71d9181a67b7542122c20230211c867954c213d408496a0c00392a91008 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/a87ff679a2f3e71d9181a67b7542122c20230211c867954c213d408496a0c00392a91008 2023-02-11 20:01:00,243 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/a87ff679a2f3e71d9181a67b7542122c20230211dcc9bcc5d07b49e88321a7e2f185f411 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/a87ff679a2f3e71d9181a67b7542122c20230211dcc9bcc5d07b49e88321a7e2f185f411 2023-02-11 20:01:00,243 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c4ca4238a0b923820dcc509a6f75849b20230211689f735cdb02427f88e276320159afad to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c4ca4238a0b923820dcc509a6f75849b20230211689f735cdb02427f88e276320159afad 2023-02-11 20:01:00,244 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c4ca4238a0b923820dcc509a6f75849b20230211bee74755269547a78094d6514813ce6e to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c4ca4238a0b923820dcc509a6f75849b20230211bee74755269547a78094d6514813ce6e 2023-02-11 20:01:00,245 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c81e728d9d4c2f636f067f89cc14862c20230211847278d687ef4e8094a419ca4751f0b4 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c81e728d9d4c2f636f067f89cc14862c20230211847278d687ef4e8094a419ca4751f0b4 2023-02-11 20:01:00,246 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c81e728d9d4c2f636f067f89cc14862c20230211fdd423b90d6c40fd8ddeeec265630199 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/c81e728d9d4c2f636f067f89cc14862c20230211fdd423b90d6c40fd8ddeeec265630199 2023-02-11 20:01:00,247 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/d41d8cd98f00b204e9800998ecf8427e20230211a615ab38ba894a7bb51b2d27eb43d491 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/d41d8cd98f00b204e9800998ecf8427e20230211a615ab38ba894a7bb51b2d27eb43d491 2023-02-11 20:01:00,248 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/d41d8cd98f00b204e9800998ecf8427e20230211db6be44d53a94924b821944cd1bc0c82 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/d41d8cd98f00b204e9800998ecf8427e20230211db6be44d53a94924b821944cd1bc0c82 2023-02-11 20:01:00,249 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/e4da3b7fbbce2345d7772b0674a318d52023021143585c2bfb1e4f68b76e5ee7d3ce2a13 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/e4da3b7fbbce2345d7772b0674a318d52023021143585c2bfb1e4f68b76e5ee7d3ce2a13 2023-02-11 20:01:00,250 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/e4da3b7fbbce2345d7772b0674a318d5202302117a42971387dc4e4abe5be69050d0dbca to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/e4da3b7fbbce2345d7772b0674a318d5202302117a42971387dc4e4abe5be69050d0dbca 2023-02-11 20:01:00,251 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211a994002e11b141b282516e1e2d1e595d to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211a994002e11b141b282516e1e2d1e595d 2023-02-11 20:01:00,251 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211f8be42215c3f4b14923eab8a227b17a6 to hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/archive/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211f8be42215c3f4b14923eab8a227b17a6 2023-02-11 20:01:00,252 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/mobdir/data/default/testCorruptedSnapshot-1676145653041/4d0b241b5252d724ef40bd367d246368 2023-02-11 20:01:00,253 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676145653041 2023-02-11 20:01:00,256 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1676145653041 from hbase:meta 2023-02-11 20:01:00,257 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1676145653041' descriptor. 2023-02-11 20:01:00,258 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676145653041 2023-02-11 20:01:00,258 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1676145653041' from region states. 2023-02-11 20:01:00,258 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145660258"}]},"ts":"9223372036854775807"} 2023-02-11 20:01:00,258 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145660258"}]},"ts":"9223372036854775807"} 2023-02-11 20:01:00,258 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145660258"}]},"ts":"9223372036854775807"} 2023-02-11 20:01:00,258 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145660258"}]},"ts":"9223372036854775807"} 2023-02-11 20:01:00,259 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145660258"}]},"ts":"9223372036854775807"} 2023-02-11 20:01:00,259 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676145660258"}]},"ts":"9223372036854775807"} 2023-02-11 20:01:00,260 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 20:01:00,260 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5bc4c474972588bf6c06dc75b6e4aea6, NAME => 'testCorruptedSnapshot-1676145653041,,1676145653044.5bc4c474972588bf6c06dc75b6e4aea6.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 7c1f214213ec2e6ba08d051362b2ca4e, NAME => 'testCorruptedSnapshot-1676145653041,1,1676145653044.7c1f214213ec2e6ba08d051362b2ca4e.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 3518e490e7f4ccb7c60b36f15d4f9183, NAME => 'testCorruptedSnapshot-1676145653041,2,1676145653044.3518e490e7f4ccb7c60b36f15d4f9183.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 2f7f00f697d9b55aa03c57a3213d5a7c, NAME => 'testCorruptedSnapshot-1676145653041,3,1676145653044.2f7f00f697d9b55aa03c57a3213d5a7c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 38dcb256366159437c4db1abc8d1671c, NAME => 'testCorruptedSnapshot-1676145653041,4,1676145653044.38dcb256366159437c4db1abc8d1671c.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 3eae0a3c58bb2f53c1289450916a28ea, NAME => 'testCorruptedSnapshot-1676145653041,5,1676145653044.3eae0a3c58bb2f53c1289450916a28ea.', STARTKEY => '5', ENDKEY => ''}] 2023-02-11 20:01:00,260 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1676145653041' as deleted. 2023-02-11 20:01:00,261 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676145653041","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676145660261"}]},"ts":"9223372036854775807"} 2023-02-11 20:01:00,262 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1676145653041 state from META 2023-02-11 20:01:00,264 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676145653041 2023-02-11 20:01:00,265 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1676145653041 in 56 msec 2023-02-11 20:01:00,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-11 20:01:00,462 INFO [Listener at localhost/33691] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1676145653041, procId: 283 completed 2023-02-11 20:01:00,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676145653041" 2023-02-11 20:01:00,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676145653041 2023-02-11 20:01:00,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676145653041" 2023-02-11 20:01:00,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39971] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676145653041 2023-02-11 20:01:00,486 INFO [Listener at localhost/33691] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=421 (was 407) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-749956425_15 at /127.0.0.1:35276 [Waiting for operation #278] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1210470265_15 at /127.0.0.1:47082 [Waiting for operation #253] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7415b278-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1610540354_15 at /127.0.0.1:36252 [Waiting for operation #694] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7415b278-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7415b278-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7415b278-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-68077062_15 at /127.0.0.1:35282 [Waiting for operation #262] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7415b278-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1210470265_15 at /127.0.0.1:35038 [Waiting for operation #9] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-749956425_15 at /127.0.0.1:40072 [Waiting for operation #504] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1210470265_15 at /127.0.0.1:36750 [Waiting for operation #199] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_530637622_15 at /127.0.0.1:47094 [Waiting for operation #240] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-68077062_15 at /127.0.0.1:36266 [Waiting for operation #732] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-68077062_15 at /127.0.0.1:35030 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6065ab53-shared-pool-34 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-749956425_15 at /127.0.0.1:40076 [Waiting for operation #532] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6065ab53-shared-pool-35 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-68077062_15 at /127.0.0.1:46848 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6065ab53-shared-pool-36 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6065ab53-shared-pool-37 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1610540354_15 at /127.0.0.1:60392 [Waiting for operation #17] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=786 (was 794), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=543 (was 556), ProcessCount=168 (was 168), AvailableMemoryMB=3073 (was 3344) 2023-02-11 20:01:00,487 INFO [Listener at localhost/33691] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-11 20:01:00,488 INFO [Listener at localhost/33691] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-11 20:01:00,488 DEBUG [Listener at localhost/33691] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b210801 to 127.0.0.1:51732 2023-02-11 20:01:00,488 DEBUG [Listener at localhost/33691] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:01:00,488 DEBUG [Listener at localhost/33691] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-11 20:01:00,488 DEBUG [Listener at localhost/33691] util.JVMClusterUtil(257): Found active master hash=466483807, stopped=false 2023-02-11 20:01:00,488 INFO [Listener at localhost/33691] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,39971,1676145618710 2023-02-11 20:01:00,492 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:44175-0x10044f8555a0003, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 20:01:00,492 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 20:01:00,492 INFO [Listener at localhost/33691] procedure2.ProcedureExecutor(629): Stopping 2023-02-11 20:01:00,492 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 20:01:00,492 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 20:01:00,492 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 20:01:00,493 DEBUG [Listener at localhost/33691] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1813e35a to 127.0.0.1:51732 2023-02-11 20:01:00,493 DEBUG [Listener at localhost/33691] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:01:00,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44175-0x10044f8555a0003, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 20:01:00,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 20:01:00,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 20:01:00,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 20:01:00,493 INFO [Listener at localhost/33691] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42011,1676145619841' ***** 2023-02-11 20:01:00,493 INFO [Listener at localhost/33691] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 20:01:00,493 INFO [Listener at localhost/33691] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35937,1676145619887' ***** 2023-02-11 20:01:00,493 INFO [Listener at localhost/33691] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 20:01:00,493 INFO [Listener at localhost/33691] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,44175,1676145619924' ***** 2023-02-11 20:01:00,494 INFO [Listener at localhost/33691] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 20:01:00,494 INFO [RS:0;jenkins-hbase4:42011] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 20:01:00,494 INFO [RS:1;jenkins-hbase4:35937] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 20:01:00,494 INFO [RS:2;jenkins-hbase4:44175] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 20:01:00,494 INFO [RS:1;jenkins-hbase4:35937] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 20:01:00,494 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 20:01:00,494 INFO [RS:1;jenkins-hbase4:35937] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 20:01:00,494 INFO [RS:0;jenkins-hbase4:42011] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 20:01:00,494 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 20:01:00,494 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 20:01:00,494 INFO [RS:2;jenkins-hbase4:44175] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 20:01:00,494 INFO [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:01:00,494 INFO [RS:0;jenkins-hbase4:42011] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 20:01:00,494 DEBUG [RS:1;jenkins-hbase4:35937] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4df21505 to 127.0.0.1:51732 2023-02-11 20:01:00,494 INFO [RS:2;jenkins-hbase4:44175] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 20:01:00,494 INFO [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer(3304): Received CLOSE for 77102a65aff4927797da7a30e0b8de4a 2023-02-11 20:01:00,494 INFO [RS:2;jenkins-hbase4:44175] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:01:00,494 DEBUG [RS:1;jenkins-hbase4:35937] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:01:00,494 DEBUG [RS:2;jenkins-hbase4:44175] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77c8214c to 127.0.0.1:51732 2023-02-11 20:01:00,495 INFO [RS:1;jenkins-hbase4:35937] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 20:01:00,495 DEBUG [RS:2;jenkins-hbase4:44175] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:01:00,495 INFO [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:01:00,495 INFO [RS:2;jenkins-hbase4:44175] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,44175,1676145619924; all regions closed. 2023-02-11 20:01:00,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 77102a65aff4927797da7a30e0b8de4a, disabling compactions & flushes 2023-02-11 20:01:00,495 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a. 2023-02-11 20:01:00,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a. 2023-02-11 20:01:00,495 INFO [RS:1;jenkins-hbase4:35937] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 20:01:00,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a. after waiting 0 ms 2023-02-11 20:01:00,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a. 2023-02-11 20:01:00,495 DEBUG [RS:0;jenkins-hbase4:42011] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ac087cc to 127.0.0.1:51732 2023-02-11 20:01:00,495 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 77102a65aff4927797da7a30e0b8de4a 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-11 20:01:00,495 INFO [RS:1;jenkins-hbase4:35937] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 20:01:00,495 DEBUG [RS:0;jenkins-hbase4:42011] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:01:00,495 INFO [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-11 20:01:00,495 INFO [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-11 20:01:00,495 DEBUG [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer(1479): Online Regions={77102a65aff4927797da7a30e0b8de4a=hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a.} 2023-02-11 20:01:00,495 INFO [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-11 20:01:00,495 DEBUG [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-02-11 20:01:00,496 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 20:01:00,496 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 20:01:00,496 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 20:01:00,496 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 20:01:00,496 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 20:01:00,496 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=137.71 KB heapSize=212.48 KB 2023-02-11 20:01:00,497 DEBUG [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-11 20:01:00,497 DEBUG [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer(1505): Waiting on 77102a65aff4927797da7a30e0b8de4a 2023-02-11 20:01:00,500 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/WALs/jenkins-hbase4.apache.org,44175,1676145619924/jenkins-hbase4.apache.org%2C44175%2C1676145619924.1676145621487 not finished, retry = 0 2023-02-11 20:01:00,507 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/namespace/77102a65aff4927797da7a30e0b8de4a/.tmp/info/7e0a3b92d37747baa8bab435e9ec91fa 2023-02-11 20:01:00,511 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=130.94 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/.tmp/info/4a31022feff840aa9c3ec3d24f7bec96 2023-02-11 20:01:00,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/namespace/77102a65aff4927797da7a30e0b8de4a/.tmp/info/7e0a3b92d37747baa8bab435e9ec91fa as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/namespace/77102a65aff4927797da7a30e0b8de4a/info/7e0a3b92d37747baa8bab435e9ec91fa 2023-02-11 20:01:00,514 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4a31022feff840aa9c3ec3d24f7bec96 2023-02-11 20:01:00,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/namespace/77102a65aff4927797da7a30e0b8de4a/info/7e0a3b92d37747baa8bab435e9ec91fa, entries=2, sequenceid=6, filesize=4.8 K 2023-02-11 20:01:00,515 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 77102a65aff4927797da7a30e0b8de4a in 20ms, sequenceid=6, compaction requested=false 2023-02-11 20:01:00,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/namespace/77102a65aff4927797da7a30e0b8de4a/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-11 20:01:00,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a. 2023-02-11 20:01:00,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 77102a65aff4927797da7a30e0b8de4a: Waiting for close lock at 1676145660495Running coprocessor pre-close hooks at 1676145660495Disabling compacts and flushes for region at 1676145660495Disabling writes for close at 1676145660495Obtaining lock to block concurrent updates at 1676145660495Preparing flush snapshotting stores in 77102a65aff4927797da7a30e0b8de4a at 1676145660495Finished memstore snapshotting hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676145660495Flushing stores of hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a. at 1676145660498 (+3 ms)Flushing 77102a65aff4927797da7a30e0b8de4a/info: creating writer at 1676145660498Flushing 77102a65aff4927797da7a30e0b8de4a/info: appending metadata at 1676145660500 (+2 ms)Flushing 77102a65aff4927797da7a30e0b8de4a/info: closing flushed file at 1676145660500Flushing 77102a65aff4927797da7a30e0b8de4a/info: reopening flushed file at 1676145660511 (+11 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 77102a65aff4927797da7a30e0b8de4a in 20ms, sequenceid=6, compaction requested=false at 1676145660515 (+4 ms)Writing region close event to WAL at 1676145660516 (+1 ms)Running coprocessor post-close hooks at 1676145660520 (+4 ms)Closed at 1676145660521 (+1 ms) 2023-02-11 20:01:00,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676145621981.77102a65aff4927797da7a30e0b8de4a. 2023-02-11 20:01:00,529 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/.tmp/rep_barrier/98a433d3a69349a2bee7e3ec8ff94ef1 2023-02-11 20:01:00,532 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 98a433d3a69349a2bee7e3ec8ff94ef1 2023-02-11 20:01:00,542 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/.tmp/table/c2f43665db004caea9cfa3621ea1c731 2023-02-11 20:01:00,545 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c2f43665db004caea9cfa3621ea1c731 2023-02-11 20:01:00,545 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/.tmp/info/4a31022feff840aa9c3ec3d24f7bec96 as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/info/4a31022feff840aa9c3ec3d24f7bec96 2023-02-11 20:01:00,547 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4a31022feff840aa9c3ec3d24f7bec96 2023-02-11 20:01:00,548 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/info/4a31022feff840aa9c3ec3d24f7bec96, entries=64, sequenceid=307, filesize=11.8 K 2023-02-11 20:01:00,548 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/.tmp/rep_barrier/98a433d3a69349a2bee7e3ec8ff94ef1 as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/rep_barrier/98a433d3a69349a2bee7e3ec8ff94ef1 2023-02-11 20:01:00,551 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 98a433d3a69349a2bee7e3ec8ff94ef1 2023-02-11 20:01:00,551 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/rep_barrier/98a433d3a69349a2bee7e3ec8ff94ef1, entries=18, sequenceid=307, filesize=7.0 K 2023-02-11 20:01:00,551 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/.tmp/table/c2f43665db004caea9cfa3621ea1c731 as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/table/c2f43665db004caea9cfa3621ea1c731 2023-02-11 20:01:00,554 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c2f43665db004caea9cfa3621ea1c731 2023-02-11 20:01:00,554 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/table/c2f43665db004caea9cfa3621ea1c731, entries=22, sequenceid=307, filesize=7.0 K 2023-02-11 20:01:00,554 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 58ms, sequenceid=307, compaction requested=false 2023-02-11 20:01:00,558 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-02-11 20:01:00,559 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-11 20:01:00,560 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 20:01:00,560 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676145660496Running coprocessor pre-close hooks at 1676145660496Disabling compacts and flushes for region at 1676145660496Disabling writes for close at 1676145660496Obtaining lock to block concurrent updates at 1676145660496Preparing flush snapshotting stores in 1588230740 at 1676145660496Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=141010, getHeapSize=217528, getOffHeapSize=0, getCellsCount=997 at 1676145660496Flushing stores of hbase:meta,,1.1588230740 at 1676145660498 (+2 ms)Flushing 1588230740/info: creating writer at 1676145660498Flushing 1588230740/info: appending metadata at 1676145660502 (+4 ms)Flushing 1588230740/info: closing flushed file at 1676145660502Flushing 1588230740/rep_barrier: creating writer at 1676145660514 (+12 ms)Flushing 1588230740/rep_barrier: appending metadata at 1676145660519 (+5 ms)Flushing 1588230740/rep_barrier: closing flushed file at 1676145660519Flushing 1588230740/table: creating writer at 1676145660532 (+13 ms)Flushing 1588230740/table: appending metadata at 1676145660534 (+2 ms)Flushing 1588230740/table: closing flushed file at 1676145660534Flushing 1588230740/info: reopening flushed file at 1676145660546 (+12 ms)Flushing 1588230740/rep_barrier: reopening flushed file at 1676145660549 (+3 ms)Flushing 1588230740/table: reopening flushed file at 1676145660551 (+2 ms)Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 58ms, sequenceid=307, compaction requested=false at 1676145660554 (+3 ms)Writing region close event to WAL at 1676145660556 (+2 ms)Running coprocessor post-close hooks at 1676145660559 (+3 ms)Closed at 1676145660560 (+1 ms) 2023-02-11 20:01:00,560 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-11 20:01:00,579 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 20:01:00,582 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 20:01:00,583 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 20:01:00,602 DEBUG [RS:2;jenkins-hbase4:44175] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/oldWALs 2023-02-11 20:01:00,602 INFO [RS:2;jenkins-hbase4:44175] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C44175%2C1676145619924:(num 1676145621487) 2023-02-11 20:01:00,602 DEBUG [RS:2;jenkins-hbase4:44175] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:01:00,602 INFO [RS:2;jenkins-hbase4:44175] regionserver.LeaseManager(133): Closed leases 2023-02-11 20:01:00,602 INFO [RS:2;jenkins-hbase4:44175] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-11 20:01:00,602 INFO [RS:2;jenkins-hbase4:44175] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 20:01:00,603 INFO [RS:2;jenkins-hbase4:44175] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 20:01:00,603 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 20:01:00,603 INFO [RS:2;jenkins-hbase4:44175] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 20:01:00,604 INFO [RS:2;jenkins-hbase4:44175] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44175 2023-02-11 20:01:00,609 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:01:00,609 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:01:00,609 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 20:01:00,609 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 20:01:00,609 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:44175-0x10044f8555a0003, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44175,1676145619924 2023-02-11 20:01:00,609 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 20:01:00,610 ERROR [Listener at localhost/33691-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@c3aa0ab rejected from java.util.concurrent.ThreadPoolExecutor@80535f4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 20:01:00,610 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:44175-0x10044f8555a0003, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 20:01:00,610 ERROR [Listener at localhost/33691-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@25e6c4ca rejected from java.util.concurrent.ThreadPoolExecutor@80535f4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 20:01:00,611 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,44175,1676145619924] 2023-02-11 20:01:00,611 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,44175,1676145619924; numProcessing=1 2023-02-11 20:01:00,613 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,44175,1676145619924 already deleted, retry=false 2023-02-11 20:01:00,613 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,44175,1676145619924 expired; onlineServers=2 2023-02-11 20:01:00,697 INFO [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35937,1676145619887; all regions closed. 2023-02-11 20:01:00,698 INFO [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42011,1676145619841; all regions closed. 2023-02-11 20:01:00,713 DEBUG [RS:0;jenkins-hbase4:42011] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/oldWALs 2023-02-11 20:01:00,713 INFO [RS:0;jenkins-hbase4:42011] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42011%2C1676145619841:(num 1676145621486) 2023-02-11 20:01:00,713 DEBUG [RS:0;jenkins-hbase4:42011] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:01:00,714 INFO [RS:0;jenkins-hbase4:42011] regionserver.LeaseManager(133): Closed leases 2023-02-11 20:01:00,714 INFO [RS:0;jenkins-hbase4:42011] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-11 20:01:00,715 DEBUG [RS:1;jenkins-hbase4:35937] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/oldWALs 2023-02-11 20:01:00,715 INFO [RS:1;jenkins-hbase4:35937] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35937%2C1676145619887.meta:.meta(num 1676145621673) 2023-02-11 20:01:00,716 INFO [RS:0;jenkins-hbase4:42011] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 20:01:00,716 INFO [RS:0;jenkins-hbase4:42011] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 20:01:00,716 INFO [RS:0;jenkins-hbase4:42011] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 20:01:00,716 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 20:01:00,717 INFO [RS:0;jenkins-hbase4:42011] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42011 2023-02-11 20:01:00,721 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:01:00,721 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 20:01:00,721 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42011,1676145619841 2023-02-11 20:01:00,722 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,42011,1676145619841] 2023-02-11 20:01:00,722 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42011,1676145619841; numProcessing=2 2023-02-11 20:01:00,725 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42011,1676145619841 already deleted, retry=false 2023-02-11 20:01:00,725 DEBUG [RS:1;jenkins-hbase4:35937] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/oldWALs 2023-02-11 20:01:00,725 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,42011,1676145619841 expired; onlineServers=1 2023-02-11 20:01:00,725 INFO [RS:1;jenkins-hbase4:35937] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35937%2C1676145619887:(num 1676145621486) 2023-02-11 20:01:00,725 DEBUG [RS:1;jenkins-hbase4:35937] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:01:00,725 INFO [RS:1;jenkins-hbase4:35937] regionserver.LeaseManager(133): Closed leases 2023-02-11 20:01:00,725 INFO [RS:1;jenkins-hbase4:35937] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 20:01:00,726 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 20:01:00,726 INFO [RS:1;jenkins-hbase4:35937] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35937 2023-02-11 20:01:00,824 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 20:01:00,824 INFO [RS:0;jenkins-hbase4:42011] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42011,1676145619841; zookeeper connection closed. 2023-02-11 20:01:00,824 ERROR [Listener at localhost/33691-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@565eb20e rejected from java.util.concurrent.ThreadPoolExecutor@52612c1d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 20:01:00,824 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:42011-0x10044f8555a0001, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 20:01:00,824 ERROR [Listener at localhost/33691-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@553279b rejected from java.util.concurrent.ThreadPoolExecutor@52612c1d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 20:01:00,824 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@36dddb3e] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@36dddb3e 2023-02-11 20:01:00,826 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35937,1676145619887 2023-02-11 20:01:00,826 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 20:01:00,826 ERROR [Listener at localhost/33691-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4aceffba rejected from java.util.concurrent.ThreadPoolExecutor@4ef1723b[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1374) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 20:01:00,828 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35937,1676145619887] 2023-02-11 20:01:00,828 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35937,1676145619887; numProcessing=3 2023-02-11 20:01:00,829 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35937,1676145619887 already deleted, retry=false 2023-02-11 20:01:00,829 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,35937,1676145619887 expired; onlineServers=0 2023-02-11 20:01:00,829 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39971,1676145618710' ***** 2023-02-11 20:01:00,829 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-11 20:01:00,830 DEBUG [M:0;jenkins-hbase4:39971] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5af0f064, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 20:01:00,830 INFO [M:0;jenkins-hbase4:39971] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39971,1676145618710 2023-02-11 20:01:00,830 INFO [M:0;jenkins-hbase4:39971] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39971,1676145618710; all regions closed. 2023-02-11 20:01:00,830 DEBUG [M:0;jenkins-hbase4:39971] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 20:01:00,830 DEBUG [M:0;jenkins-hbase4:39971] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-11 20:01:00,831 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-11 20:01:00,831 DEBUG [M:0;jenkins-hbase4:39971] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-11 20:01:00,831 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676145621085] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676145621085,5,FailOnTimeoutGroup] 2023-02-11 20:01:00,831 INFO [M:0;jenkins-hbase4:39971] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-11 20:01:00,831 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676145621083] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676145621083,5,FailOnTimeoutGroup] 2023-02-11 20:01:00,831 INFO [M:0;jenkins-hbase4:39971] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-11 20:01:00,832 INFO [M:0;jenkins-hbase4:39971] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-11 20:01:00,832 DEBUG [M:0;jenkins-hbase4:39971] master.HMaster(1502): Stopping service threads 2023-02-11 20:01:00,832 INFO [M:0;jenkins-hbase4:39971] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-11 20:01:00,832 ERROR [M:0;jenkins-hbase4:39971] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-11 20:01:00,833 INFO [M:0;jenkins-hbase4:39971] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-11 20:01:00,833 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-11 20:01:00,834 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-11 20:01:00,834 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 20:01:00,834 DEBUG [M:0;jenkins-hbase4:39971] zookeeper.ZKUtil(398): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-11 20:01:00,834 WARN [M:0;jenkins-hbase4:39971] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-11 20:01:00,834 INFO [M:0;jenkins-hbase4:39971] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-11 20:01:00,834 INFO [M:0;jenkins-hbase4:39971] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-11 20:01:00,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 20:01:00,834 DEBUG [M:0;jenkins-hbase4:39971] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 20:01:00,834 INFO [M:0;jenkins-hbase4:39971] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 20:01:00,834 DEBUG [M:0;jenkins-hbase4:39971] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 20:01:00,834 DEBUG [M:0;jenkins-hbase4:39971] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 20:01:00,834 DEBUG [M:0;jenkins-hbase4:39971] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 20:01:00,834 INFO [M:0;jenkins-hbase4:39971] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=843.54 KB heapSize=1.01 MB 2023-02-11 20:01:00,852 INFO [M:0;jenkins-hbase4:39971] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=843.54 KB at sequenceid=2048 (bloomFilter=true), to=hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ddb02940d46d4ec38347e77f63def060 2023-02-11 20:01:00,854 INFO [M:0;jenkins-hbase4:39971] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ddb02940d46d4ec38347e77f63def060 2023-02-11 20:01:00,855 DEBUG [M:0;jenkins-hbase4:39971] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ddb02940d46d4ec38347e77f63def060 as hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ddb02940d46d4ec38347e77f63def060 2023-02-11 20:01:00,857 INFO [M:0;jenkins-hbase4:39971] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ddb02940d46d4ec38347e77f63def060 2023-02-11 20:01:00,857 INFO [M:0;jenkins-hbase4:39971] regionserver.HStore(1080): Added hdfs://localhost:43583/user/jenkins/test-data/250b76e5-5222-0bde-8c98-a32b2a4b6ff5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ddb02940d46d4ec38347e77f63def060, entries=282, sequenceid=2048, filesize=30.7 K 2023-02-11 20:01:00,858 INFO [M:0;jenkins-hbase4:39971] regionserver.HRegion(2947): Finished flush of dataSize ~843.54 KB/863788, heapSize ~1.01 MB/1055480, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 24ms, sequenceid=2048, compaction requested=false 2023-02-11 20:01:00,859 INFO [M:0;jenkins-hbase4:39971] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 20:01:00,859 DEBUG [M:0;jenkins-hbase4:39971] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676145660834Disabling compacts and flushes for region at 1676145660834Disabling writes for close at 1676145660834Obtaining lock to block concurrent updates at 1676145660834Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676145660834Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=863788, getHeapSize=1055480, getOffHeapSize=0, getCellsCount=2535 at 1676145660835 (+1 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676145660835Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676145660835Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676145660840 (+5 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676145660840Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676145660855 (+15 ms)Finished flush of dataSize ~843.54 KB/863788, heapSize ~1.01 MB/1055480, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 24ms, sequenceid=2048, compaction requested=false at 1676145660858 (+3 ms)Writing region close event to WAL at 1676145660859 (+1 ms)Closed at 1676145660859 2023-02-11 20:01:00,863 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 20:01:00,863 INFO [M:0;jenkins-hbase4:39971] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-11 20:01:00,863 INFO [M:0;jenkins-hbase4:39971] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39971 2023-02-11 20:01:00,865 DEBUG [M:0;jenkins-hbase4:39971] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,39971,1676145618710 already deleted, retry=false 2023-02-11 20:01:01,092 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 20:01:01,092 INFO [M:0;jenkins-hbase4:39971] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39971,1676145618710; zookeeper connection closed. 2023-02-11 20:01:01,092 ERROR [Listener at localhost/33691-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3f33a4f rejected from java.util.concurrent.ThreadPoolExecutor@3fa73eef[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 20:01:01,092 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): master:39971-0x10044f8555a0000, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 20:01:01,092 ERROR [Listener at localhost/33691-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@38ce3721 rejected from java.util.concurrent.ThreadPoolExecutor@3fa73eef[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 20:01:01,192 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 20:01:01,192 INFO [RS:1;jenkins-hbase4:35937] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35937,1676145619887; zookeeper connection closed. 2023-02-11 20:01:01,192 ERROR [Listener at localhost/33691-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7bc76f5 rejected from java.util.concurrent.ThreadPoolExecutor@4ef1723b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 20:01:01,192 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:35937-0x10044f8555a0002, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 20:01:01,192 ERROR [Listener at localhost/33691-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7f17a750 rejected from java.util.concurrent.ThreadPoolExecutor@4ef1723b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 20:01:01,192 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5f63a4d5] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5f63a4d5 2023-02-11 20:01:01,292 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:44175-0x10044f8555a0003, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 20:01:01,292 INFO [RS:2;jenkins-hbase4:44175] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,44175,1676145619924; zookeeper connection closed. 2023-02-11 20:01:01,292 ERROR [Listener at localhost/33691-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5d8b37d4 rejected from java.util.concurrent.ThreadPoolExecutor@80535f4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 20:01:01,293 DEBUG [Listener at localhost/33691-EventThread] zookeeper.ZKWatcher(600): regionserver:44175-0x10044f8555a0003, quorum=127.0.0.1:51732, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 20:01:01,293 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1b4bee0a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1b4bee0a 2023-02-11 20:01:01,293 ERROR [Listener at localhost/33691-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3efb0424 rejected from java.util.concurrent.ThreadPoolExecutor@80535f4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 20:01:01,293 INFO [Listener at localhost/33691] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-11 20:01:01,293 WARN [Listener at localhost/33691] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 20:01:01,297 INFO [Listener at localhost/33691] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 20:01:01,401 WARN [BP-902064344-172.31.14.131-1676145615035 heartbeating to localhost/127.0.0.1:43583] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 20:01:01,401 WARN [BP-902064344-172.31.14.131-1676145615035 heartbeating to localhost/127.0.0.1:43583] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-902064344-172.31.14.131-1676145615035 (Datanode Uuid 6172956c-8b21-46b3-8a3e-55f2d221c88d) service to localhost/127.0.0.1:43583 2023-02-11 20:01:01,403 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748/dfs/data/data5/current/BP-902064344-172.31.14.131-1676145615035] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 20:01:01,403 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748/dfs/data/data6/current/BP-902064344-172.31.14.131-1676145615035] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 20:01:01,405 WARN [Listener at localhost/33691] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 20:01:01,407 INFO [Listener at localhost/33691] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 20:01:01,512 WARN [BP-902064344-172.31.14.131-1676145615035 heartbeating to localhost/127.0.0.1:43583] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 20:01:01,512 WARN [BP-902064344-172.31.14.131-1676145615035 heartbeating to localhost/127.0.0.1:43583] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-902064344-172.31.14.131-1676145615035 (Datanode Uuid c1a54257-2d0a-45a8-ba80-a5d503ce840b) service to localhost/127.0.0.1:43583 2023-02-11 20:01:01,513 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748/dfs/data/data3/current/BP-902064344-172.31.14.131-1676145615035] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 20:01:01,513 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748/dfs/data/data4/current/BP-902064344-172.31.14.131-1676145615035] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 20:01:01,514 WARN [Listener at localhost/33691] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 20:01:01,516 INFO [Listener at localhost/33691] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 20:01:01,621 WARN [BP-902064344-172.31.14.131-1676145615035 heartbeating to localhost/127.0.0.1:43583] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 20:01:01,621 WARN [BP-902064344-172.31.14.131-1676145615035 heartbeating to localhost/127.0.0.1:43583] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-902064344-172.31.14.131-1676145615035 (Datanode Uuid 6c4dbc0d-7a4b-4326-b825-e2c9ec77a985) service to localhost/127.0.0.1:43583 2023-02-11 20:01:01,621 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748/dfs/data/data1/current/BP-902064344-172.31.14.131-1676145615035] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 20:01:01,622 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/28f43ad8-4a79-857e-f878-f939a05a8524/cluster_4e641da9-f839-127f-014f-354d78021748/dfs/data/data2/current/BP-902064344-172.31.14.131-1676145615035] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 20:01:01,650 INFO [Listener at localhost/33691] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 20:01:01,778 INFO [Listener at localhost/33691] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-11 20:01:01,882 INFO [Listener at localhost/33691] hbase.HBaseTestingUtility(1293): Minicluster is down